Detecting physiological responses while taking into account consumption of confounding substances

ABSTRACT

Described herein are systems and methods for detecting a physiological response based on thermal measurements while accounting for consumption of a confounding substance such as a medication, alcohol, caffeine, or nicotine. In one embodiment, a system includes a computer and an inward-facing head-mounted thermal camera (CAM) that takes thermal measurements of a region of interest (TH ROI ) on a user&#39;s face. The computer receives an indication indicative of the user consuming a confounding substance that affects TH ROI , and detects the physiological response while the consumed confounding substance affects TH ROI . The detection is based on TH ROI , the indication and a model. Optionally, the model was trained on: a first set of TH ROI  taken while the confounding substance affected TH ROI , and a second set of TH ROI  taken while the confounding substance did not affect TH ROI .

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application claims priority to U.S. Provisional Patent ApplicationNo. 62/456,105, filed Feb. 7, 2017, and U.S. Provisional PatentApplication No. 62/480,496, filed Apr. 2, 2017, and U.S. ProvisionalPatent Application No. 62/566,572, filed Oct. 2, 2017. U.S. ProvisionalPatent Application No. 62/566,572 is herein incorporated by reference inits entirety.

This Application is a Continuation-In-Part of U.S. application Ser. No.15/182,566, filed Jun. 14, 2016, now U.S. Pat. No. [TBD], which claimspriority to U.S. Provisional Patent Application No. 62/175,319, filedJun. 14, 2015, and U.S. Provisional Patent Application No. 62/202,808,filed Aug. 8, 2015.

This Application is also a Continuation-In-Part of U.S. application Ser.No. 15/182,592, filed Jun. 14, 2016, which claims priority to U.S.Provisional Patent Application No. 62/175,319, filed Jun. 14, 2015, andU.S. Provisional Patent Application No. 62/202,808, filed Aug. 8, 2015.

This Application is also a Continuation-In-Part of U.S. application Ser.No. 15/231,276, filed Aug. 8, 2016, which claims priority to U.S.Provisional Patent Application No. 62/202,808, filed Aug. 8, 2015, andU.S. Provisional Patent Application No. 62/236,868, filed Oct. 3, 2015.

This Application is also a Continuation-In-Part of U.S. application Ser.No. 15/284,528, filed Oct. 3, 2016, which claims priority to U.S.Provisional Patent Application No. 62/236,868, filed Oct. 3, 2015, andU.S. Provisional Patent Application No. 62/354,833, filed Jun. 27, 2016,and U.S. Provisional Patent Application No. 62/372,063, filed Aug. 8,2016.

This Application is also a Continuation-In-Part of U.S. application Ser.No. 15/635,178, filed Jun. 27, 2017, which claims priority to U.S.Provisional Patent Application No. 62/354,833, filed Jun. 27, 2016, andU.S. Provisional Patent Application No. 62/372,063, filed Aug. 8, 2016.

This Application is also a Continuation-In-Part of U.S. application Ser.No. 15/722,434, filed Oct. 2, 2017, which claims priority to U.S.Provisional Patent Application No. 62/408,677, filed Oct. 14, 2016, andU.S. Provisional Patent Application No. 62/456,105, filed Feb. 7, 2017,and U.S. Provisional Patent Application No. 62/480,496, filed Apr. 2,2017. U.S. Ser. No. 15/722,434 is also a Continuation-In-Part of U.S.application Ser. No. 15/182,592, filed Jun. 14, 2016, which claimspriority to U.S. Provisional Patent Application No. 62/175,319, filedJun. 14, 2015, and U.S. Provisional Patent Application No. 62/202,808,filed Aug. 8, 2015. U.S. Ser. No. 15/722,434 is also aContinuation-In-Part of U.S. application Ser. No. 15/231,276, filed Aug.8, 2016, which claims priority to U.S. Provisional Patent ApplicationNo. 62/202,808, filed Aug. 8, 2015, and U.S. Provisional PatentApplication No. 62/236,868, filed Oct. 3, 2015. And U.S. Ser. No.15/722,434 is also a Continuation-In-Part of U.S. application Ser. No.15/284,528, filed Oct. 3, 2016, which claims priority to U.S.Provisional Patent Application No. 62/236,868, filed Oct. 3, 2015, andU.S. Provisional Patent Application No. 62/354,833, filed Jun. 27, 2016,and U.S. Provisional Patent Application No. 62/372,063, filed Aug. 8,2016.

ACKNOWLEDGMENTS

Gil Thieberger would like to thank his holy and beloved teacher, LamaDvora-hla, for her extraordinary teachings and manifestation of wisdom,love, compassion and morality, and for her endless efforts, support, andskills in guiding him and others on their paths to freedom and ultimatehappiness. Gil would also like to thank his beloved parents for raisinghim exactly as they did.

BACKGROUND

Thermal measurements of the face can be utilized for variousapplications, which include detection of various physiological responsesand/or medical conditions. However, when taken in uncontrolled real-lifescenarios, thermal measurements can be influenced by various intrinsicand extrinsic factors. Some examples of such factors include infraredradiation directed at the face (e.g., sunlight), wind, physical contactwith the face (e.g., touching the face with the hand), and consumptionof substances such as various medications, alcohol, caffeine, andnicotine. Though these factors may be unrelated to a physiologicalresponse and/or medical condition being detected, they may nonethelessaffect the temperatures at various regions of interest on the face.Therefore, these factors may be considered confounding factors thathinder the detection of the physiological response and/or medicalcondition. Thus, there is a need for systems that utilize thermalmeasurements collected in real-life scenarios to account for variousintrinsic and/or extrinsic factors that may alter the thermalmeasurements, in order to avoid the detrimental effect that such factorsmay have on the utility of the thermal measurements for variousapplications.

SUMMARY

Collecting thermal measurements of various regions of a user's face canhave many health-related (and other) applications, such as identifyingvarious physiological responses of the user. However, the values of themeasurements may be influenced, in some cases, by intrinsic factors,such as consumption of substances such as various medications, alcohol,caffeine, and nicotine. Consumption of the aforementioned substances,which may affect the values of the measurements, is unrelated to theuser's condition and can affect, in some cases, the accuracy ofdetections of physiological responses made based on thermal measurementsof the face. Consequently, the substances may be referred to herein as“confounding substances”. Some aspects of this disclosure describesystems and/or methods in which the effects of consumption ofconfounding substances may be mitigated, which may improve the accuracyof detections of the physiological responses and/or medical conditionsmade based on thermal measurements of the face.

One aspect of this disclosure involves a system that is configured todetect a physiological response while taking into account consumption ofa confounding substance, which includes an inward-facing head-mountedthermal camera (CAM) and a computer. CAM takes thermal measurements of aregion of interest (TH_(ROI)) on a user's face. The computer receives anindication indicative of the user consuming a confounding substance thataffects TH_(ROI), and detects the physiological response while theconsumed confounding substance affects TH_(ROI). The detection isperformed based on TH_(ROI), the indication, and a model. Optionally,the model was trained on: a first set of TH_(ROI) taken while theconfounding substance affected TH_(ROI), and a second set of TH_(ROI)taken while the confounding substance did not affect TH_(ROI).Optionally, the first and second sets are previous TH_(ROI) of the user.Optionally, the model is a machine learning-based model, and thecomputer generates feature values based on TH_(ROI) and the indication,and utilizes the machine learning-based model to detect thephysiological response based on the feature values.

Another aspect of this disclosure includes a method for detecting aphysiological response while taking into account consumption of aconfounding substance, which includes the following steps: In Step 1,taking, using an inward-facing head-mounted thermal camera, thermalmeasurements of a region of interest (TH_(ROI)) on a user's face. InStep 2, receiving an indication indicative of consuming a confoundingsubstance that affects TH_(ROI). And in Step 3, detecting thephysiological response, while the consumed confounding substance affectsTH_(ROI), based on TH_(ROI), the indication, and a model. Optionally,the model was trained on: a first set of TH_(ROI) taken while theconfounding substance affected TH_(ROI), and a second set of TH_(ROI)taken while the confounding substance did not affect TH_(ROI).Optionally, the model is a machine learning-based model, and the methodincludes steps involving: generating feature values based on TH_(ROI)and the indication, and utilizing the machine learning-based model todetect the physiological response based on the feature values.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described by way of example only, withreference to the following drawings:

FIG. 1a and FIG. 1b illustrate various inward-facing head-mountedcameras coupled to an eyeglasses frame;

FIG. 2 illustrates inward-facing head-mounted cameras coupled to anaugmented reality device;

FIG. 3 illustrates head-mounted cameras coupled to a virtual realitydevice;

FIG. 4 illustrates a side view of head-mounted cameras coupled to anaugmented reality device;

FIG. 5 illustrates a side view of head-mounted cameras coupled to asunglasses frame;

FIG. 6, FIG. 7, FIG. 8 and FIG. 9 illustrate head-mounted systems (HMSs)configured to measure various ROIs relevant to some of the embodimentsdescribes herein;

FIG. 10, FIG. 11, FIG. 12 and FIG. 13 illustrate various embodiments ofsystems that include inward-facing head-mounted cameras havingmulti-pixel sensors (FPA sensors);

FIG. 14a , FIG. 14b , and FIG. 14c illustrate embodiments of two rightand left clip-on devices that are configured to attached/detached froman eyeglasses frame;

FIG. 15a and FIG. 15b illustrate an embodiment of a clip-on device thatincludes inward-facing head-mounted cameras pointed at the lower part ofthe face and the forehead;

FIG. 16a and FIG. 16b illustrate embodiments of right and left clip-ondevices that are configured to be attached behind an eyeglasses frame;

FIG. 17a and FIG. 17b illustrate an embodiment of a single-unit clip-ondevice that is configured to be attached behind an eyeglasses frame;

FIG. 18 illustrates embodiments of right and left clip-on devices, whichare configured to be attached/detached from an eyeglasses frame, andhave protruding arms to hold inward-facing head-mounted cameras;

FIG. 19 illustrates a scenario in which an alert regarding a possiblestroke is issued;

FIG. 20a is a schematic illustration of an inward-facing head-mountedcamera embedded in an eyeglasses frame, which utilizes the Scheimpflugprinciple;

FIG. 20b is a schematic illustration of a camera that is able to changethe relative tilt between its lens and sensor planes according to theScheimpflug principle;

FIG. 21a and FIG. 21b illustrate the making of different detections ofemotional response based on thermal measurements compared to theemotional response that is visible in a facial expression;

FIG. 22 illustrates an embodiment of a smartphone app that provides auser with feedback about how he/she looks to others;

FIG. 23 illustrates one embodiment of a tablet app that provides theuser a feedback about how he/she felt during a certain period;

FIG. 24 illustrates an embodiment of the system configured to detect aphysiological response based on facial skin color changes (FSCC);

FIG. 25a and FIG. 25b illustrate heating of a ROI for different reasons:sinusitis (which is detected), and acne (which is not detected assinusitis);

FIG. 26a and FIG. 26b illustrate an embodiment of a system that providesindications when the user touches his/her face;

FIG. 27a illustrates a first case where a user's hair does not occludethe forehead;

FIG. 27b illustrates a second case where a user's hair occludes theforehead and the system requests the user to move the hair in order toenable correct measurements of the forehead;

FIG. 28a illustrates an embodiment of a system that detects aphysiological response based on measurements taken by an inward-facinghead-mounted thermal camera and an outward-facing head-mounted thermalcamera;

FIG. 28b illustrates a scenario in which a user receives an indicationon a GUI that the user is not monitored in direct sunlight;

FIG. 29 illustrates a case in which a user receives an indication thatshe is not being monitored in a windy environment;

FIG. 30 illustrates an elderly person whose facial temperature increasesas a result of bending over;

FIG. 31 illustrates the effect of consuming alcohol on values of thermalmeasurements;

FIG. 32 illustrates an increase in the thermal measurements due tosmoking;

FIG. 33 illustrates a decrease in the thermal measurements due to takingmedication; and

FIG. 34a and FIG. 34b are schematic illustrations of possibleembodiments for computers.

DETAILED DESCRIPTION

A “thermal camera” refers herein to a non-contact device that measureselectromagnetic radiation having wavelengths longer than 2500 nanometer(nm) and does not touch its region of interest (ROI). A thermal cameramay include one sensing element (pixel), or multiple sensing elementsthat are also referred to herein as “sensing pixels”, “pixels”, and/orfocal-plane array (FPA). A thermal camera may be based on an uncooledthermal sensor, such as a thermopile sensor, a microbolometer sensor(where microbolometer refers to any type of a bolometer sensor and itsequivalents), a pyroelectric sensor, or a ferroelectric sensor.

Sentences in the form of “thermal measurements of an ROI” (usuallydenoted TH_(ROI) or some variant thereof) refer to at least one of: (i)temperature measurements of the ROI (T_(ROI)), such as when usingthermopile or microbolometer sensors, and (ii) temperature changemeasurements of the ROI (ΔT_(ROI)), such as when using a pyroelectricsensor or when deriving the temperature changes from temperaturemeasurements taken at different times by a thermopile sensor or amicrobolometer sensor.

In some embodiments, a device, such as a thermal camera, may bepositioned such that it occludes an ROI on the user's face, while inother embodiments, the device may be positioned such that it does notocclude the ROI. Sentences in the form of “the system/camera does notocclude the ROI” indicate that the ROI can be observed by a third personlocated in front of the user and looking at the ROI, such as illustratedby all the ROIs in FIG. 7, FIG. 11 and FIG. 19. Sentences in the form of“the system/camera occludes the ROI” indicate that some of the ROIscannot be observed directly by that third person, such as ROIs 19 and 37that are occluded by the lenses in FIG. 1a , and ROIs 97 and 102 thatare occluded by cameras 91 and 96, respectively, in FIG. 9.

Although many of the disclosed embodiments can use occluding thermalcameras successfully, in certain scenarios, such as when using an HMS ona daily basis and/or in a normal day-to-day setting, using thermalcameras that do not occlude their ROIs on the face may provide one ormore advantages to the user, to the HMS, and/or to the thermal cameras,which may relate to one or more of the following: esthetics, betterventilation of the face, reduced weight, simplicity to wear, and reducedlikelihood to being tarnished.

A “Visible-light camera” refers to a non-contact device designed todetect at least some of the visible spectrum, such as a camera withoptical lenses and CMOS or CCD sensor.

The term “inward-facing head-mounted camera” refers to a cameraconfigured to be worn on a user's head and to remain pointed at its ROI,which is on the user's face, also when the user's head makes angular andlateral movements (such as movements with an angular velocity above 0.1rad/sec, above 0.5 rad/sec, and/or above 1 rad/sec). A head-mountedcamera (which may be inward-facing and/or outward-facing) may bephysically coupled to a frame worn on the user's head, may be attachedto eyeglass using a clip-on mechanism (configured to be attached to anddetached from the eyeglasses), or may be mounted to the user's headusing any other known device that keeps the camera in a fixed positionrelative to the user's head also when the head moves. Sentences in theform of “camera physically coupled to the frame” mean that the cameramoves with the frame, such as when the camera is fixed to (or integratedinto) the frame, or when the camera is fixed to (or integrated into) anelement that is physically coupled to the frame. The abbreviation “CAM”denotes “inward-facing head-mounted thermal camera”, the abbreviation“CAM_(out)” denotes “outward-facing head-mounted thermal camera”, theabbreviation “VCAM” denotes “inward-facing head-mounted visible-lightcamera”, and the abbreviation “VCAM_(out)” denotes “outward-facinghead-mounted visible-light camera”.

Sentences in the form of “a frame configured to be worn on a user'shead” or “a frame worn on a user's head” refer to a mechanical structurethat loads more than 50% of its weight on the user's head. For example,an eyeglasses frame may include two temples connected to two rimsconnected by a bridge; the frame in Oculus Rift™ includes the foamplaced on the user's face and the straps; and the frames in GoogleGlass™ and Spectacles by Snap Inc. are similar to eyeglasses frames.Additionally or alternatively, the frame may connect to, be affixedwithin, and/or be integrated with, a helmet (e.g., sports, motorcycle,bicycle, and/or combat helmets) and/or a brainwave-measuring headset.

When a thermal camera is inward-facing and head-mounted, challengesfaced by systems known in the art that are used to acquire thermalmeasurements, which include non-head-mounted thermal cameras, may besimplified and even eliminated with some of the embodiments describedherein. Some of these challenges may involve dealing with complicationscaused by movements of the user, image registration, ROI alignment,tracking based on hot spots or markers, and motion compensation in theIR domain.

In various embodiments, cameras are located close to a user's face, suchas at most 2 cm, 5 cm, 10 cm, 15 cm, or 20 cm from the face (herein “cm”denotes to centimeters). The distance from the face/head in sentencessuch as “a camera located less than 15 cm from the face/head” refers tothe shortest possible distance between the camera and the face/head. Thehead-mounted cameras used in various embodiments may be lightweight,such that each camera weighs below 10 g, 5 g, 1 g, and/or 0.5 g (herein“g” denotes to grams).

The following figures show various examples of HMSs equipped withhead-mounted cameras. FIG. 1a illustrates various inward-facinghead-mounted cameras coupled to an eyeglasses frame 15. Cameras 10 and12 measure regions 11 and 13 on the forehead, respectively. Cameras 18and 36 measure regions on the periorbital areas 19 and 37, respectively.The HMS further includes an optional computer 16, which may include aprocessor, memory, a battery and/or a communication module. FIG. 1billustrates a similar HMS in which inward-facing head-mounted cameras 48and 49 measure regions 41 and 41, respectively. Cameras 22 and 24measure regions 23 and 25, respectively. Camera 28 measures region 29.And cameras 26 and 43 measure regions 38 and 39, respectively.

FIG. 2 illustrates inward-facing head-mounted cameras coupled to anaugmented reality device such as Microsoft HoloLens™. FIG. 3 illustrateshead-mounted cameras coupled to a virtual reality device such asFacebook's Oculus Rift™. FIG. 4 is a side view illustration ofhead-mounted cameras coupled to an augmented reality device such asGoogle Glass™. FIG. 5 is another side view illustration of head-mountedcameras coupled to a sunglasses frame.

FIG. 6 to FIG. 9 illustrate HMSs configured to measure various ROIsrelevant to some of the embodiments describes herein. FIG. 6 illustratesa frame 35 that mounts inward-facing head-mounted cameras 30 and 31 thatmeasure regions 32 and 33 on the forehead, respectively. FIG. 7illustrates a frame 75 that mounts inward-facing head-mounted cameras 70and 71 that measure regions 72 and 73 on the forehead, respectively, andinward-facing head-mounted cameras 76 and 77 that measure regions 78 and79 on the upper lip, respectively. FIG. 8 illustrates a frame 84 thatmounts inward-facing head-mounted cameras 80 and 81 that measure regions82 and 83 on the sides of the nose, respectively. And FIG. 9 illustratesa frame 90 that includes (i) inward-facing head-mounted cameras 91 and92 that are mounted to protruding arms and measure regions 97 and 98 onthe forehead, respectively, (ii) inward-facing head-mounted cameras 95and 96, which are also mounted to protruding arms, which measure regions101 and 102 on the lower part of the face, respectively, and (iii)head-mounted cameras 93 and 94 that measure regions on the periorbitalareas 99 and 100, respectively.

FIG. 10 to FIG. 13 illustrate various inward-facing head-mounted camerashaving multi-pixel sensors (FPA sensors), configured to measure variousROIs relevant to some of the embodiments describes herein. FIG. 10illustrates head-mounted cameras 120 and 122 that measure regions 121and 123 on the forehead, respectively, and mounts head-mounted camera124 that measure region 125 on the nose. FIG. 11 illustrateshead-mounted cameras 126 and 128 that measure regions 127 and 129 on theupper lip, respectively, in addition to the head-mounted cameras alreadydescribed in FIG. 10. FIG. 12 illustrates head-mounted cameras 130 and132 that measure larger regions 131 and 133 on the upper lip and thesides of the nose, respectively. And FIG. 13 illustrates head-mountedcameras 134 and 137 that measure regions 135 and 138 on the right andleft cheeks and right and left sides of the mouth, respectively, inaddition to the head-mounted cameras already described in FIG. 12.

In some embodiments, the head-mounted cameras may be physically coupledto the frame using a clip-on device configured to be attached/detachedfrom a pair of eyeglasses in order to secure/release the device to/fromthe eyeglasses, multiple times. The clip-on device holds at least aninward-facing camera, a processor, a battery, and a wirelesscommunication module. Most of the clip-on device may be located in frontof the frame (as illustrated in FIG. 14b , FIG. 15b , and FIG. 18), oralternatively, most of the clip-on device may be located behind theframe, as illustrated in FIG. 16b and FIG. 17 b.

FIG. 14a , FIG. 14b , and FIG. 14c illustrate two right and left clip-ondevices 141 and 142, respectively, configured to attached/detached froman eyeglasses frame 140. The clip-on device 142 includes aninward-facing head-mounted camera 143 pointed at a region on the lowerpart of the face (such as the upper lip, mouth, nose, and/or cheek), aninward-facing head-mounted camera 144 pointed at the forehead, and otherelectronics 145 (such as a processor, a battery, and/or a wirelesscommunication module). The clip-on devices 141 and 142 may includeadditional cameras illustrated in the drawings as black circles.

FIG. 15a and FIG. 15b illustrate a clip-on device 147 that includes aninward-facing head-mounted camera 148 pointed at a region on the lowerpart of the face (such as the nose), and an inward-facing head-mountedcamera 149 pointed at the forehead. The other electronics (such as aprocessor, a battery, and/or a wireless communication module) is locatedinside the box 150, which also holds the cameras 148 and 149.

FIG. 16a and FIG. 16b illustrate two right and left clip-on devices 160and 161, respectively, configured to be attached behind an eyeglassesframe 165. The clip-on device 160 includes an inward-facing head-mountedcamera 162 pointed at a region on the lower part of the face (such asthe upper lip, mouth, nose, and/or cheek), an inward-facing head-mountedcamera 163 pointed at the forehead, and other electronics 164 (such as aprocessor, a battery, and/or a wireless communication module). Theclip-on devices 160 and 161 may include additional cameras illustratedin the drawings as black circles.

FIG. 17a and FIG. 17b illustrate a single-unit clip-on device 170,configured to be attached behind an eyeglasses frame 176. Thesingle-unit clip-on device 170 includes inward-facing head-mountedcameras 171 and 172 pointed at regions on the lower part of the face(such as the upper lip, mouth, nose, and/or cheek), inward-facinghead-mounted cameras 173 and 174 pointed at the forehead, a spring 175configured to apply force that holds the clip-on device 170 to the frame176, and other electronics 177 (such as a processor, a battery, and/or awireless communication module). The clip-on device 170 may includeadditional cameras illustrated in the drawings as black circles.

FIG. 18 illustrates two right and left clip-on devices 153 and 154,respectively, configured to attached/detached from an eyeglasses frame,and having protruding arms to hold the inward-facing head-mountedcameras. Head-mounted camera 155 measures a region on the lower part ofthe face, head-mounted camera 156 measures regions on the forehead, andthe left clip-on device 154 further includes other electronics 157 (suchas a processor, a battery, and/or a wireless communication module). Theclip-on devices 153 and 154 may include additional cameras illustratedin the drawings as black circles.

It is noted that the elliptic and other shapes of the ROIs in some ofthe drawings are just for illustration purposes, and the actual shapesof the ROIs are usually not as illustrated. It is possible to calculatethe accurate shape of an ROI using various methods, such as acomputerized simulation using a 3D model of the face and a model of ahead-mounted system (HMS) to which a thermal camera is physicallycoupled, or by placing a LED instead of the sensor (while maintainingthe same field of view) and observing the illumination pattern on theface. Furthermore, illustrations and discussions of a camera representone or more cameras, where each camera may have the same FOV and/ordifferent FOVs. Unless indicated to the contrary, the cameras mayinclude one or more sensing elements (pixels), even when multiplesensing elements do not explicitly appear in the figures; when a cameraincludes multiple sensing elements then the illustrated ROI usuallyrefers to the total ROI captured by the camera, which is made ofmultiple regions that are respectively captured by the different sensingelements. The positions of the cameras in the figures are just forillustration, and the cameras may be placed at other positions on theHMS.

Sentences in the form of an “ROI on an area”, such as ROI on theforehead or an ROI on the nose, refer to at least a portion of the area.Depending on the context, and especially when using a CAM having justone pixel or a small number of pixels, the ROI may cover another area(in addition to the area). For example, a sentence in the form of “anROI on the nose” may refer to either: 100% of the ROI is on the nose, orsome of the ROI is on the nose and some of the ROI is on the upper lip.

Various embodiments described herein involve detections of physiologicalresponses based on user measurements. Some examples of physiologicalresponses include stress, an allergic reaction, an asthma attack, astroke, dehydration, intoxication, or a headache (which includes amigraine). Other examples of physiological responses includemanifestations of fear, startle, sexual arousal, anxiety, joy, pain orguilt. Still other examples of physiological responses includephysiological signals such as a heart rate or a value of a respiratoryparameter of the user. Optionally, detecting a physiological responsemay involve one or more of the following: determining whether the userhas/had the physiological response, identifying an imminent attackassociated with the physiological response, and/or calculating theextent of the physiological response.

In some embodiments, detection of the physiological response is done byprocessing thermal measurements that fall within a certain window oftime that characterizes the physiological response. For example,depending on the physiological response, the window may be five secondslong, thirty seconds long, two minutes long, five minutes long, fifteenminutes long, or one hour long. Detecting the physiological response mayinvolve analysis of thermal measurements taken during multiple of theabove-described windows, such as measurements taken during differentdays. In some embodiments, a computer may receive a stream of thermalmeasurements, taken while the user wears an HMS with coupled thermalcameras during the day, and periodically evaluate measurements that fallwithin a sliding window of a certain size.

In some embodiments, models are generated based on measurements takenover long periods. Sentences of the form of “measurements taken duringdifferent days” or “measurements taken over more than a week” are notlimited to continuous measurements spanning the different days or overthe week, respectively. For example, “measurements taken over more thana week” may be taken by eyeglasses equipped with thermal cameras, whichare worn for more than a week, 8 hours a day. In this example, the useris not required to wear the eyeglasses while sleeping in order to takemeasurements over more than a week. Similarly, sentences of the form of“measurements taken over more than 5 days, at least 2 hours a day” referto a set comprising at least 10 measurements taken over 5 differentdays, where at least two measurements are taken each day at timesseparated by at least two hours.

Utilizing measurements taken of a long period (e.g., measurements takenon “different days”) may have an advantage, in some embodiments, ofcontributing to the generalizability of a trained model. Measurementstaken over the long period likely include measurements taken indifferent environments and/or measurements taken while the measured userwas in various physiological and/or mental states (e.g., before/aftermeals and/or while the measured user wassleepy/energetic/happy/depressed, etc.). Training a model on such datacan improve the performance of systems that utilize the model in thediverse settings often encountered in real-world use (as opposed tocontrolled laboratory-like settings). Additionally, taking themeasurements over the long period may have the advantage of enablingcollection of a large amount of training data that is required for somemachine learning approaches (e.g., “deep learning”).

Detecting the physiological response may involve performing varioustypes of calculations by a computer. Optionally, detecting thephysiological response may involve performing one or more of thefollowing operations: comparing thermal measurements to a threshold(when the threshold is reached that may be indicative of an occurrenceof the physiological response), comparing thermal measurements to areference time series, and/or by performing calculations that involve amodel trained using machine learning methods. Optionally, the thermalmeasurements upon which the one or more operations are performed aretaken during a window of time of a certain length, which may optionallydepend on the type of physiological response being detected. In oneexample, the window may be shorter than one or more of the followingdurations: five seconds, fifteen seconds, one minute, five minutes,thirty minute, one hour, four hours, one day, or one week. In anotherexample, the window may be longer than one or more of the aforementioneddurations. Thus, when measurements are taken over a long period, such asmeasurements taken over a period of more than a week, detection of thephysiological response at a certain time may be done based on a subsetof the measurements that falls within a certain window near the certaintime; the detection at the certain time does not necessarily involveutilizing all values collected throughout the long period.

In some embodiments, detecting the physiological response of a user mayinvolve utilizing baseline thermal measurement values, most of whichwere taken when the user was not experiencing the physiologicalresponse. Optionally, detecting the physiological response may rely onobserving a change to typical temperatures at one or more ROIs (thebaseline), where different users might have different typicaltemperatures at the ROIs (i.e., different baselines). Optionally,detecting the physiological response may rely on observing a change to abaseline level, which is determined based on previous measurements takenduring the preceding minutes and/or hours.

In some embodiments, detecting a physiological response involvesdetermining the extent of the physiological response, which may beexpressed in various ways that are indicative of the extent of thephysiological response, such as: (i) a binary value indicative ofwhether the user experienced, and/or is experiencing, the physiologicalresponse, (ii) a numerical value indicative of the magnitude of thephysiological response, (iii) a categorial value indicative of theseverity/extent of the physiological response, (iv) an expected changein thermal measurements of an ROI (denoted TH_(ROI) or some variationthereof), and/or (v) rate of change in TH_(ROI). Optionally, when thephysiological response corresponds to a physiological signal (e.g., aheart rate, a breathing rate, and an extent of frontal lobe brainactivity), the extent of the physiological response may be interpretedas the value of the physiological signal.

One approach for detecting a physiological response, which may beutilized in some embodiments, involves comparing thermal measurements ofone or more ROIs to a threshold. In these embodiments, the computer maydetect the physiological response by comparing the thermal measurements,and/or values derived therefrom (e.g., a statistic of the measurementsand/or a function of the measurements), to the threshold to determinewhether it is reached. Optionally, the threshold may include a thresholdin the time domain, a threshold in the frequency domain, an upperthreshold, and/or a lower threshold. When a threshold involves a certainchange to temperature, the certain change may be positive (increase intemperature) or negative (decrease in temperature). Differentphysiological responses described herein may involve different types ofthresholds, which may be an upper threshold (where reaching thethreshold means≧the threshold) or a lower threshold (where reaching thethreshold means≦the threshold); for example, each physiological responsemay involve at least a certain degree of heating, or at least a certaindegree cooling, at a certain ROI on the face.

Another approach for detecting a physiological response, which may beutilized in some embodiments, may be applicable when the thermalmeasurements of a user are treated as time series data. For example, thethermal measurements may include data indicative of temperatures at oneor more ROIs at different points of time during a certain period. Insome embodiments, the computer may compare thermal measurements(represented as a time series) to one or more reference time series thatcorrespond to periods of time in which the physiological responseoccurred. Additionally or alternatively, the computer may compare thethermal measurements to other reference time series corresponding totimes in which the physiological response did not occur. Optionally, ifthe similarity between the thermal measurements and a reference timeseries corresponding to a physiological response reaches a threshold,this is indicative of the fact that the thermal measurements correspondto a period of time during which the user had the physiologicalresponse. Optionally, if the similarity between the thermal measurementsand a reference time series that does not correspond to a physiologicalresponse reaches another threshold, this is indicative of the fact thatthe thermal measurements correspond to a period of time in which theuser did not have the physiological response. Time series analysis mayinvolve various forms of processing involving segmenting data, aligningdata, clustering, time warping, and various functions for determiningsimilarity between sequences of time series data. Some of the techniquesthat may be utilized in various embodiments are described in Ding, Hui,et al. “Querying and mining of time series data: experimental comparisonof representations and distance measures.” Proceedings of the VLDBEndowment 1.2 (2008): 1542-1552, and in Wang, Xiaoyue, et al.“Experimental comparison of representation methods and distance measuresfor time series data.” Data Mining and Knowledge Discovery 26.2 (2013):275-309.

Herein, “machine learning” methods refers to learning from examplesusing one or more approaches. Optionally, the approaches may beconsidered supervised, semi-supervised, and/or unsupervised methods.Examples of machine learning approaches include: decision tree learning,association rule learning, regression models, nearest neighborsclassifiers, artificial neural networks, deep learning, inductive logicprogramming, support vector machines, clustering, Bayesian networks,reinforcement learning, representation learning, similarity and metriclearning, sparse dictionary learning, genetic algorithms, rule-basedmachine learning, and/or learning classifier systems.

Herein, a “machine learning-based model” is a model trained usingmachine learning methods. For brevity's sake, at times, a “machinelearning-based model” may simply be called a “model”. Referring to amodel as being “machine learning-based” is intended to indicate that themodel is trained using machine learning methods (otherwise, “model” mayalso refer to a model generated by methods other than machine learning).

In some embodiments, which involve utilizing a machine learning-basedmodel, a computer is configured to detect the physiological response bygenerating feature values based on the thermal measurements (andpossibly other values), and/or based on values derived therefrom (e.g.,statistics of the measurements). The computer then utilizes the machinelearning-based model to calculate, based on the feature values, a valuethat is indicative of whether, and/or to what extent, the user isexperiencing (and/or is about to experience) the physiological response.Optionally, calculating said value is considered “detecting thephysiological response”. Optionally, the value calculated by thecomputer is indicative of the probability that the user has/had thephysiological response.

Herein, feature values may be considered input to a computer thatutilizes a model to perform the calculation of a value, such as thevalue indicative of the extent of the physiological response mentionedabove. It is to be noted that the terms “feature” and “feature value”may be used interchangeably when the context of their use is clear.However, a “feature” typically refers to a certain type of value, andrepresents a property, while “feature value” is the value of theproperty with a certain instance (sample). For example, a feature may betemperature at a certain ROI, while the feature value corresponding tothat feature may be 36.9° C. in one instance and 37.3° C. in anotherinstance.

In some embodiments, a machine learning-based model used to detect aphysiological response is trained based on data that includes samples.Each sample includes feature values and a label. The feature values mayinclude various types of values. At least some of the feature values ofa sample are generated based on measurements of a user taken during acertain period of time (e.g., thermal measurements taken during thecertain period of time). Optionally, some of the feature values may bebased on various other sources of information described herein. Thelabel is indicative of a physiological response of the usercorresponding to the certain period of time. Optionally, the label maybe indicative of whether the physiological response occurred during thecertain period and/or the extent of the physiological response duringthe certain period. Additionally or alternatively, the label may beindicative of how long the physiological response lasted. Labels ofsamples may be generated using various approaches, such as self-reportby users, annotation by experts that analyze the training data,automatic annotation by a computer that analyzes the training dataand/or analyzes additional data related to the training data, and/orutilizing additional sensors that provide data useful for generating thelabels. It is to be noted that herein when it is stated that a model istrained based on certain measurements (e.g., “a model trained based onTH_(ROI) taken on different days”), it means that the model was trainedon samples comprising feature values generated based on the certainmeasurements and labels corresponding to the certain measurements.Optionally, a label corresponding to a measurement is indicative of thephysiological response at the time the measurement was taken.

Various types of feature values may be generated based on thermalmeasurements. In one example, some feature values are indicative oftemperatures at certain ROIs. In another example, other feature valuesmay represent a temperature change at certain ROIs. The temperaturechanges may be with respect to a certain time and/or with respect to adifferent ROI. In order to better detect physiological responses thattake some time to manifest, in some embodiments, some feature values maydescribe temperatures (or temperature changes) at a certain ROI atdifferent points of time. Optionally, these feature values may includevarious functions and/or statistics of the thermal measurements such asminimum/maximum measurement values and/or average values during certainwindows of time.

It is to be noted that when it is stated that feature values aregenerated based on data comprising multiple sources, it means that foreach source, there is at least one feature value that is generated basedon that source (and possibly other data). For example, stating thatfeature values are generated from thermal measurements of first andsecond ROIs (TH_(ROI1) and TH_(ROI2), respectively) means that thefeature values may include a first feature value generated based onTH_(ROI1) and a second feature value generated based on TH_(ROI2).Optionally, a sample is considered generated based on measurements of auser (e.g., measurements comprising TH_(ROI1) and TH_(ROI2)) when itincludes feature values generated based on the measurements of the user.

In addition to feature values that are generated based on thermalmeasurements, in some embodiments, at least some feature values utilizedby a computer (e.g., to detect a physiological response or train a mode)may be generated based on additional sources of data that may affecttemperatures measured at various facial ROIs. Some examples of theadditional sources include: (i) measurements of the environment such astemperature, humidity level, noise level, elevation, air quality, a windspeed, precipitation, and infrared radiation; (ii) contextualinformation such as the time of day (e.g., to account for effects of thecircadian rhythm), day of month (e.g., to account for effects of thelunar rhythm), day in the year (e.g., to account for seasonal effects),and/or stage in a menstrual cycle; (iii) information about the userbeing measured such as sex, age, weight, height, and/or body build.Alternatively or additionally, at least some feature values may begenerated based on physiological signals of the user obtained by sensorsthat are not thermal cameras, such as a visible-light camera, aphotoplethysmogram (PPG) sensor, an electrocardiogram (ECG) sensor, anelectroencephalography (EEG) sensor, a galvanic skin response (GSR)sensor, or a thermistor.

The machine learning-based model used to detect a physiological responsemay be trained, in some embodiments, based on data collected inday-to-day, real world scenarios. As such, the data may be collected atdifferent times of the day, while users perform various activities, andin various environmental conditions. Utilizing such diverse trainingdata may enable a trained model to be more resilient to the variouseffects different conditions can have on the values of thermalmeasurements, and consequently, be able to achieve better detection ofthe physiological response in real world day-to-day scenarios.

Since real world day-to-day conditions are not the same all the time,sometimes detection of the physiological response may be hampered bywhat is referred to herein as “confounding factors”. A confoundingfactor can be a cause of warming and/or cooling of certain regions ofthe face, which is unrelated to a physiological response being detected,and as such, may reduce the accuracy of the detection of thephysiological response. Some examples of confounding factors include:(i) environmental phenomena such as direct sunlight, air conditioning,and/or wind; (ii) things that are on the user's face, which are nottypically there and/or do not characterize the faces of most users(e.g., cosmetics, ointments, sweat, hair, facial hair, skin blemishes,acne, inflammation, piercings, body paint, and food leftovers); (iii)physical activity that may affect the user's heart rate, bloodcirculation, and/or blood distribution (e.g., walking, running, jumping,and/or bending over); (iv) consumption of substances to which the bodyhas a physiological response that may involve changes to temperatures atvarious facial ROIs, such as various medications, alcohol, caffeine,tobacco, and/or certain types of food; and/or (v) disruptive facialmovements (e.g., frowning, talking, eating, drinking, sneezing, andcoughing).

Occurrences of confounding factors may not always be easily identifiedin thermal measurements. Thus, in some embodiments, systems mayincorporate measures designed to accommodate for the confoundingfactors. In some embodiments, these measures may involve generatingfeature values that are based on additional sensors, other than thethermal cameras. In some embodiments, these measures may involverefraining from detecting the physiological response, which should beinterpreted as refraining from providing an indication that the user hasthe physiological response. For example, if an occurrence of a certainconfounding factor is identified, such as strong directional sunlightthat heats one side of the face, the system may refrain from detectingthat the user had a stroke. In this example, the user may not be alertedeven though a temperature difference between symmetric ROIs on bothsides of the face reaches a threshold that, under other circumstances,would warrant alerting the user.

Training data used to train a model for detecting a physiologicalresponse may include, in some embodiments, a diverse set of samplescorresponding to various conditions, some of which involve occurrence ofconfounding factors (when there is no physiological response and/or whenthere is a physiological response). Having samples in which aconfounding factor occurs (e.g., the user is in direct sunlight ortouches the face) can lead to a model that is less susceptible towrongfully detect the physiological response (which may be considered anoccurrence of a false positive) in real world situations.

When a model is trained with training data comprising samples generatedfrom measurements of multiple users, the model may be considered ageneral model. When a model is trained with training data comprising atleast a certain proportion of samples generated from measurements of acertain user, and/or when the samples generated from the measurements ofthe certain user are associated with at least a certain proportion ofweight in the training data, the model may be considered a personalizedmodel for the certain user. Optionally, the personalized model for thecertain user provides better results for the certain user, compared to ageneral model that was not personalized for the certain user.Optionally, personalized model may be trained based on measurements ofthe certain user, which were taken while the certain user was indifferent situations; for example, train the model based on measurementstaken while the certain user had a headache/epilepsy/stress/angerattack, and while the certain user did not have said attack.Additionally or alternatively, the personalized model may be trainedbased on measurements of the certain user, which were taken over aduration long enough to span different situations; examples of such longenough durations may include: a week, a month, six months, a year, andthree years.

Training a model that is personalized for a certain user may requirecollecting a sufficient number of training samples that are generatedbased on measurements of the certain user. Thus, initially detecting thephysiological response with the certain user may be done utilizing ageneral model, which may be replaced by a personalized model for thecertain user, as a sufficiently large number of samples are generatedbased on measurements of the certain user. Another approach involvesgradually modifying a general model based on samples of the certain userin order to obtain the personalized model.

After a model is trained, the model may be provided for use by a systemthat detects the physiological response. Providing the model may involveperforming different operations. In one embodiment, providing the modelto the system involves forwarding the model to the system via a computernetwork and/or a shared computer storage medium (e.g., writing the modelto a memory that may be accessed by the system that detects thephysiological response). In another embodiment, providing the model tothe system involves storing the model in a location from which thesystem can retrieve the model, such as a database and/or cloud-basedstorage from which the system may retrieve the model. In still anotherembodiment, providing the model involves notifying the system regardingthe existence of the model and/or regarding an update to the model.Optionally, this notification includes information needed in order forthe system to obtain the model.

A model for detecting a physiological response may include differenttypes of parameters. Following are some examples of variouspossibilities for the model and the type of calculations that may beaccordingly performed by a computer in order to detect the physiologicalresponse: (a) the model comprises parameters of a decision tree.Optionally, the computer simulates a traversal along a path in thedecision tree, determining which branches to take based on the featurevalues. A value indicative of the physiological response may be obtainedat the leaf node and/or based on calculations involving values on nodesand/or edges along the path; (b) the model comprises parameters of aregression model (e.g., regression coefficients in a linear regressionmodel or a logistic regression model). Optionally, the computermultiplies the feature values (which may be considered a regressor) withthe parameters of the regression model in order to obtain the valueindicative of the physiological response; and/or (c) the model comprisesparameters of a neural network. For example, the parameters may includevalues defining at least the following: (i) an interconnection patternbetween different layers of neurons, (ii) weights of theinterconnections, and (iii) activation functions that convert eachneuron's weighted input to its output activation. Optionally, thecomputer provides the feature values as inputs to the neural network,computes the values of the various activation functions and propagatesvalues between layers, and obtains an output from the network, which isthe value indicative of the physiological response.

A user interface (UI) may be utilized, in some embodiments, to notifythe user and/or some other entity, such as a caregiver, about thephysiological response and/or present an alert responsive to anindication that the extent of the physiological response reaches athreshold. The UI may include a screen to display the notificationand/or alert, a speaker to play an audio notification, a tactile UI,and/or a vibrating UI. In some embodiments, “alerting” about aphysiological response of a user refers to informing about one or moreof the following: the occurrence of a physiological response that theuser does not usually have (e.g., a stroke, intoxication, and/ordehydration), an imminent physiological response (e.g., an allergicreaction, an epilepsy attack, and/or a migraine), and an extent of thephysiological response reaching a threshold (e.g., stress and/or angerreaching a predetermined level).

FIG. 19 illustrates a scenario in which an alert regarding a possiblestroke is issued. The figure illustrates a user wearing a frame with atleast two CAMs (562 and 563) for measuring ROIs on the right and leftcheeks (ROIs 560 and 561, respectively). The measurements indicate thatthe left side of the face is colder than the right side of the face.Based on these measurements, and possibly additional data, the systemdetects the stroke and issues an alert. Optionally, the user's facialexpression is slightly distorted and asymmetric, and a VCAM providesadditional data in the form of images that may help detecting thestroke.

Various physiological responses may be detected based on thermalmeasurements and images of various regions of the face. In oneembodiment, a system configured to detect a physiological responseincludes an inward-facing head-mounted thermal camera (CAM), aninward-facing head-mounted visible-light camera (VCAM), and a computer.The system may optionally include additional elements such as a frameand additional cameras.

CAM is worn on a user's head and takes thermal measurements of a firstROI (TH_(ROI1)) on the user's face. Optionally, CAM weighs below 10 g.Optionally, CAM is located less than 15 cm from the user's face.Optionally, CAM utilizes a microbolometer or a thermopile sensor. In oneembodiment, CAM includes a focal-plane array (FPA) sensor and aninfrared lens, and the FPA plane is tilted by more than 2° relative tothe infrared lens plane according to the Scheimpflug principle in orderto improve the sharpness of the image of ROI₁ (where the lens planerefers to a plane that is perpendicular to the optical axis of the lens,which may include one or more lenses).

VCAM is worn on the user's head and takes images of a second ROI(IM_(ROI2)) on the user's face. Optionally, VCAM weighs below 10 g andis located less than 15 cm from the face. Optionally, ROI₁ and ROI₂overlap (which means extend over so as to cover at least partly). Forexample, ROI₂ may cover at least half of the area covered by ROI₁. Inone embodiment, VCAM includes a multi-pixel sensor and a lens, and thesensor plane is tilted by more than 2° relative to the lens planeaccording to the Scheimpflug principle in order to improve the sharpnessof the image of ROI₂.

It is to be noted that in some embodiments the system may be constructedin a way that none of the system's components (including the frame andcameras) occludes ROI₁ and/or ROI₂. In alternative embodiments, thesystem may be constructed in a way that at least some of the systemcomponents (e.g., the frame and/or CAM) may occlude ROI₁ and/or ROI₂.

The computer detects the physiological response based on TH_(ROI1),IM_(ROI2), and a model. Optionally, the model includes one or morethresholds to which TH_(ROI1) and/or IM_(ROI2) may be compared in orderto detect the physiological response. Optionally, the model includes oneor more reference time series to which TH_(ROI1) and/or IM_(ROI2) may becompared in order to detect the physiological response. Optionally, thecomputer detects the physiological response by generating feature valuesbased on TH_(ROI1) and IM_(ROI2), and utilizing the model to calculate,based on the feature values, a value indicative of the extent of thephysiological response. In this case, the model may be referred to as a“machine learning-based model”. Optionally, at least some of the featurevalues, which are generated based on IM_(ROI2) may be used to identify,and/or account for, various confounding factors that can alter TH_(ROI1)without being directly related to the physiological response. Thus, onaverage, detections of the physiological responses based on TH_(ROI1)and IM_(ROI2) are more accurate than detections of the physiologicalresponses based on TH_(ROI1) without IM_(ROI2).

In one example, the physiological response is indicative of anoccurrence of at least one of the following emotional states of theuser: joy, fear, sadness, and anger. In another example, thephysiological response is indicative of an occurrence of one or more ofthe following: stress, mental workload, an allergic reaction, aheadache, dehydration, intoxication, and a stroke. The physiologicalresponse may be a physiological signal of the user. In one example, thephysiological response is a heart rate of the user, and in this example,ROI₁ is on the skin above at least one of the superficial temporalartery and the frontal superficial temporal artery. In another example,the physiological response is frontal lobe brain activity of the user,and in this example, ROI₁ is on the forehead. In still another example,the physiological signal is a breathing rate of the user, and ROI₁ is onthe nasal area.

A machine learning-based model used to detect a physiological responseis typically trained on samples, where each sample includes featurevalues generated based on TH_(ROI1) and IM_(ROI2) taken during a certainperiod, and a label indicative of the physiological response of the userduring the certain period. Optionally, the model is trained on samplesgenerated based on measurements of the user (in which case the model maybe considered a personalized model of the user). Optionally, the modelis trained on samples generated based on measurements of one or moreother users. Optionally, the samples are generated based on measurementstaken while the user being measured was in different situations.Optionally, the samples are generated based on measurements taken ondifferent days.

In some embodiments, images such as IM_(ROI2) may be utilized togenerate various types of feature values, which may be utilized todetect the physiological response and/or detect an occurrence of aconfounding factor. Some of the feature values generated based on imagesmay include high-level facial-related feature values and theirderivatives, such as location and dimensions of facial features and/orlandmarks, identification of action units (AUs) in sequences of images,and/or blendshape weights. Other examples of features include variouslow-level features such as features generated using Gabor filters, localbinary patterns (LBP) and their derivatives, algorithms such as SIFTand/or SURF (and their derivatives), image keypoints, histograms oforiented gradients (HOG) descriptors, and statistical procedures suchindependent component analysis (ICA), principal component analysis(PCA), or linear discriminant analysis (LDA). Yet other examples offeature values may include features derived from multiple images takenat different times, such as volume local binary patterns (VLBP),cuboids, and/or optical strain-based features. Additionally, some of thefeature values may be based on other data, such as feature valuesgenerated based audio processing of data received from a head-mountedmicrophone. The audio processing may detect noises associated withtalking, eating, and drinking, and convert it to feature values to beprovided to the machine learning-based model.

Using both TH_(ROI1) and IM_(ROI2) to detect the physiological responsemay confer some advantages in some embodiments. For example, there maybe times when TH_(ROI1) and IM_(ROI2) provide complementing signals of aphysiological response (e.g., due to their ability to measuremanifestations of different physiological processes related to thephysiological response). This can increase the accuracy of thedetections. In one embodiment, in which the physiological response beingdetected is an emotional response, the computer may identify facialexpressions from IM_(ROI2), and detect the emotional response of theuser based on TH_(ROI1) and the identified facial expressions. Forexample, at least some of the feature values generated based onIM_(ROI2), which are used to detect the emotional response, areindicative of the facial expressions. Optionally, on average, detectionsof emotional responses based on both TH_(ROI1) and the identified facialexpressions are more accurate than detections of the emotional responsesbased on either TH_(ROI1) or the identified facial expressions.

The following are some specific examples how IM_(ROI2) may be utilizedto help make detections of a physiological response more accurate. Inone example, ROI₁ and ROI₂ are on the mouth, and IM_(ROI2) areindicative of a change in a facial expression during a certain periodthat involves a transition from a facial expression in which the lipsare in contact to a facial expression with an open mouth. Optionally, byutilizing IM_(ROI2) to detect the physiological response based onTH_(ROI1) taken during the certain period, the computer may be ableattribute a change in TH_(ROI1) to opening the mouth rather than achange in the temperature of the lips.

In another example, ROI₁ and ROI₂ are on the nose and upper lip, andIM_(ROI2) are indicative of a change in a facial expression during acertain period that involves a transition from a neutral facialexpression to a facial expression of disgust. Optionally, by utilizingIM_(ROI2) to detect the physiological response based on TH_(ROI1) takenduring the certain period, the computer may be able attribute a changein TH_(ROI1) to a raised upper lip and wrinkled nose instead of a changein the temperature of the nose and upper lip.

In yet another example, ROI₁ and ROI₂ are on the user's forehead locatedabout 1 cm above at least one of the user's eyebrows, and IM_(ROI2) areindicative of a change in a facial expression during a certain periodthat involves a transition from a neutral expression to a facialexpression involving raised and/or lowered eyebrows (includingmiddle-raised or middle-lowered eyebrows). Optionally, by utilizingIM_(ROI2) to detect the physiological response based on TH_(ROI1) takenduring the certain period, the computer may be able attribute a changein TH_(ROI1) to raising and/or lowering the eyebrows instead of a changein the temperature of the forehead.

It is to be noted that there are various approaches known in the art foridentifying facial expressions from images. While many of theseapproaches were originally designed for full-face frontal images, thoseskilled in the art will recognize that algorithms designed for full-facefrontal images may be easily adapted to be used with images obtainedusing the inward-facing head-mounted visible-light cameras disclosedherein. For example, the various machine learning techniques describedin prior art references may be applied to feature values extracted fromimages that include portions of the face from orientations that are notdirectly in front of the user. Furthermore, due to the closeness of VCAMto the face, facial features are typically larger in images obtained bythe systems described herein. Moreover, challenges such as imageregistration and face tracking are vastly simplified and possiblynon-existent when using inward-facing head-mounted cameras. Thereference Zeng, Zhihong, et al. “A survey of affect recognition methods:Audio, visual, and spontaneous expressions.” IEEE transactions onpattern analysis and machine intelligence 31.1 (2009): 39-58, describessome of the algorithmic approaches that may be used for this task.

In some embodiments, TH_(ROI1) and IM_(ROI2) may provide different andeven possibly contradicting indications regarding the physiologicalresponse. In particular, facial expressions may not always express how auser truly feels. For example, when in company of other people, a usermay conceal his or her true feelings by making non-genuine facialexpressions. However, at the same time, thermal measurements of theuser's face may reveal the user's true emotions. Thus, a system thatrelies only on IM_(ROI2) to determine the user's emotional response maybe mistaken at times, and using TH_(ROI1) can help make detections moreaccurate.

In one example, responsive to receiving a first set of TH_(ROI1) andIM_(ROI2) taken during a first period in which the user expressed acertain facial expression, the computer detects a first emotionalresponse of the user. Additionally, responsive to receiving a second setof TH_(ROI1) and IM_(ROI2) taken during a second period in which theuser expressed again the certain facial expression, the computer detectsa second emotional response of the user, which is not the same as thefirst emotional response. The computer detected different emotionalresponses in this example because TH_(ROI1) of the first set areindicative of a first physiological response, while TH_(ROI1) of thesecond set are indicative of a second physiological response. Followingare some more detailed examples of situations in which this may occur.

In one example, the first set includes IM_(ROI2) indicative of a facialexpression that is a smile and TH_(ROI1) indicative of stress below acertain threshold, and the first emotional response detected by thecomputer is happiness. The second set in this example includes IM_(ROI2)indicative of a facial expression that is a smile and TH_(ROI1)indicative of stress above the certain threshold, and the secondemotional response detected by the computer is discomfort.

In another example, the first set includes IM_(ROI2) indicative of afacial expression that is a neutral expression and TH_(ROI1) indicativeof stress below a certain threshold, and the first emotional responsedetected by the computer is comfort. The second set includes IM_(ROI2)indicative of a facial expression that is neutral and TH_(ROI1)indicative of stress above the certain threshold, and the secondemotional response detected by the computer is concealment.

In yet another example, the first set includes IM_(ROI2) indicative of afacial expression that is an expression of anger and TH_(ROI1)indicative of stress above a certain threshold, and the first emotionalresponse detected by the computer is anger. The second set includesIM_(ROI2) indicative of a facial expression that is an expression ofanger and TH_(ROI1) indicative of stress below the certain threshold,and the second emotional response detected by the computer is indicativeof pretending to be angry.

The phenomenon of making different detections based on thermalmeasurements compared to the emotional response that is visible in afacial expression is illustrated in FIG. 21a and FIG. 21b . Theillustrated figures include an HMS with CAM 514 and VCAM 515 that maycover portions of a cheek, mouth and/or nose. FIG. 21a illustrates acase in which the user's smiling face may be mistaken for happiness;however, the cold nose indicates that the user is in fact stressed. FIG.21b illustrates a case in which the facial expression indicates that theuser is in a neutral state; however, the warm nose indicates that theuser is excited. FIG. 21a and FIG. 21b also illustrate a second CAM 516and a second VCAM 517, which may be utilized in some embodiments, asdescribed herein.

FIG. 22 illustrates one embodiment of a smartphone app that provides theuser a feedback about how he/she looks to others. The illustrated appshows that the user was happy 96 time and angry 20 times. Because thepurpose of this app is to measure how the user looks to others, thecomputer counts the facial expressions based on IM_(ROI2) withoutcorrecting the facial expressions according TH_(ROI1).

FIG. 23 illustrates one embodiment of a tablet app that provides theuser a feedback about how he/she felt during a certain period (e.g.,during the day, the week, or while being at a certain location). Theillustrated app shows that the user felt sad 56 minutes and happy 135minutes. Because the purpose of this app is to measure how the userfeels (and not just how the user looks to others), the computerdetermines the user's emotional state based on a combined analysis ofIM_(ROI2) and TH_(ROI1), as exemplified above.

In one embodiment, the system may include a second inward-facinghead-mounted thermal camera (CAM2) that takes thermal measurements of athird ROI (TH_(ROI3)) on the face. Optionally, CAM2 weighs below 10 gand is physically coupled to the frame. Optionally, the center of ROI₁is to the right of the center of the third region of interest (ROI₃),and the symmetric overlapping between ROI₁ and ROI₃ is above 50%.Optionally, to detect the physiological response, the computer accountsfor facial thermal asymmetry, based on a difference between TH_(ROI1)and TH_(ROI3).

It is noted that the symmetric overlapping is considered with respect tothe vertical symmetry axis that divides the face to the right and leftportions. The symmetric overlapping between ROI₁ and ROI₃ may beobserved by comparing the overlap between ROI₁ and a mirror image ofROI₃, where the mirror image is with respect to a mirror that isperpendicular to the front of the face and whose intersection with theface is along the vertical symmetry axis (which goes through the middleof the forehead and the middle of the nose).

Some examples of calculations that may be performed by the computer toaccount for thermal asymmetry include: (i) utilizing differentthresholds to which TH_(ROI1) and TH_(ROI1) are compared; (ii) utilizingdifferent reference time series to which TH_(ROI1) and TH_(ROI3) arecompared; (iii) utilizing a machine learning-based model that providesdifferent results for first and second events that involve the sameaverage change in TH_(ROI1) and TH_(ROI3) with different extents ofasymmetry in TH_(ROI1) and TH_(ROI3); and (iv) utilizing the asymmetryfor differentiating between (a) temperature changes in TH_(ROI1) andTH_(ROI3) that are related to the physiological response and (b)temperature changes in TH_(ROI1) and TH_(ROI3) that are unrelated to thephysiological response.

In one embodiment, the system may include a second inward-facinghead-mounted visible-light camera (VCAM2) that takes images of a thirdROI (IM_(ROI3)) on the face. Optionally, VCAM2 weighs below 10 g and isphysically coupled to the frame. Optionally, VCAM and VCAM2 are locatedat least 0.5 cm to the right and to the left of the vertical symmetryaxis that divides the face, respectively, and the symmetric overlappingbetween ROI₂ and ROI₃ is above 50%. Optionally, the computer detects thephysiological response also based on IM_(ROI3). For example, thecomputer may generate some feature values based on IM_(ROI3), which maybe similar to feature values generated based on IM_(ROI2), and utilizesthe some feature values in the detection of the physiological response.In another example, the computer detects the physiological responsebased on the extent of symmetry between symmetric facial elementsextracted from IM_(ROI2) and IM_(ROI2).

In some embodiments, IM_(ROI2) may include recognizable facial skincolor changes (FSCC). FSCC are typically a result of changes in theconcentration levels of hemoglobin and blood oxygenation under a user'sfacial skin, and are discussed in more detail elsewhere in thisdisclosure. In one embodiment, the computer calculates, based onIM_(ROI2), a value indicative of FSCC, and detects an emotional state ofthe user based on the calculated value. Optionally, on average,detections of the physiological response based on both TH_(ROI1) andFSCC are more accurate than detections of the physiological responsebased on either TH_(ROI1) or FSCC. In another embodiment, the computergenerates feature values that are indicative of FSCC in IM_(ROI2), andutilizes a model to detect the physiological response based on thefeature values. Optionally, at least some of the feature values aregenerated based on TH_(ROI1). Optionally, the model was trained based onsamples, with each sample including feature values generated based oncorresponding measurements of the user and a label indicative of thephysiological response. Optionally, the label may be derived, forexample, from analysis of the user's speech/writing, facial expressionanalysis, speech emotion analysis, and/or emotion extraction fromanalyzing galvanic skin response (GSR) and heart rate variability (HRV).

IM_(ROI2) may be utilized, in some embodiments, to detect occurrences ofconfounding factors that can affect the temperature on the face, but areunrelated to the physiological response being detected. Thus,occurrences of confounding factors can reduce the accuracy of detectionsof the physiological response based on thermal measurements (such asbased on TH_(ROI1)). Detecting occurrences of the confounding factorsdescribed below (cosmetics, sweat, hair, inflammation and touching) maybe done utilizing various image-processing and/or image-analysistechniques known in the art. For example, detecting occurrences of atleast some of the confounding factors described below may involve amachine learning algorithm trained to detect the confounding factors,and/or comparing IM_(ROI2) to reference images that involve and do notinvolve the confounding factor (e.g., a first set of reference IM_(ROI2)in which makeup was applied to the face and a second set of referenceIM_(ROI2) in which the face was bare of makeup).

The computer may utilize detection of confounding factors in variousways in order to improve the detection of the physiological responsebased on TH_(ROI1). In one embodiment, the computer may refrain frommaking a detection of the physiological response responsive toidentifying that the extent of a certain confounding factor reaches athreshold. For example, certain physiological responses may not bedetected if there is extensive facial hair on the face or extensive skininflammation. In another embodiment, the model used to detect thephysiological response may include a certain feature that corresponds toa certain confounding factor, and the computer may generate a certainfeature value indicative of the extent of the certain confoundingfactor. Optionally, the model in this case may be trained on samples inwhich the certain feature has different values, such as some of thesamples used to train the model are generated based on measurementstaken while the certain confounding factor occurred, and other samplesused to train the model were generated based on measurements taken whilethe certain confounding factor did not occur. In yet another embodiment,the computer may weight measurements based on the occurrence ofconfounding factors, such that measurements taken while certainconfounding factors occurred, may be given lower weights thanmeasurements taken while the certain confounding factor did not occur.Optionally, lower weights for measurements mean that they have a smallerinfluence on the detection of the physiological response thanmeasurements with higher weights. The following are some examples ofconfounding factors that may be detected, in some embodiments, based onIM_(ROI2).

Some types of cosmetics (e.g., makeup and/or cream) may mask an ROI,affect the ROI's emissivity, and/or affect the ROI's temperature. Thus,taking into account cosmetics as a confounding factor may improve thesystem's ability to detect the physiological response. In oneembodiment, the model was trained on: samples generated based on a firstset of TH_(ROI1) and IM_(ROI2) taken after cosmetics were applied to aportion of the overlapping region between ROI₁ and ROI₂, and othersamples generated based on a second set of TH_(ROI1) and IM_(ROI2) takenwhile the overlapping region was bare of cosmetics. Optionally,utilizing this model may enable the computer to account for presence ofcosmetics on a portion of ROI₂.

Sweating may affect the ROI's emissivity. Thus, taking into accountsweating as a confounding factor may improve the system's ability todetect the physiological response. In one embodiment, the model wastrained on: samples generated from a first set of TH_(ROI1) andIM_(ROI2) taken while sweat was detectable on a portion of theoverlapping region between ROI1 and ROI2, and additional samplesgenerated from a second set of TH_(ROI1) and IM_(ROI2) taken while sweatwas not detectable on the overlapping region. Optionally, utilizing thismodel may enable the computer to account for sweat on the overlappingregion.

Dense hair may affect the ROI's emissivity, which may make the ROIappear, in thermal imaging, colder than it really is. Thus, taking intoaccount hair density and/or hair length (both referred to as hairdensity) as a confounding factor may improve the system's ability todetect the physiological response. In one embodiment, the model wastrained on: samples generated from a first set of TH_(ROI1) andIM_(ROI2) taken while hair density on a portion of the overlappingregion between ROI₁ and ROI₂ was at a first level, and additionalsamples generated from a second set of TH_(ROI1) and IM_(ROI2) takenwhile hair density on the portion of the overlapping region between ROI₁and ROI₂ was at a second level higher than the first level. Optionally,utilizing a model trained so may enable the computer to account for hairon the overlapping region.

In another embodiment, when the hair can be moved the system may requestthe user to move her hair in order to enable the thermal cameras to takecorrect measurements. For example, FIG. 27a illustrates a first casewhere the user's hair does not occlude the forehead. FIG. 27billustrates a second case where the user's hair does occlude theforehead and thus the system requests the user to move the hair in orderto enable correct measurements of the forehead.

Skin inflammations (which may include skin blemishes, acne, and/orinflammatory skin diseases) usually increases ROI temperature in amanner that is unrelated to the physiological response being detected.Thus, taking into account skin inflammation as a confounding factor mayimprove the system's ability to detect the physiological response. FIG.25a illustrates heating of the ROI because of sinusitis, for which thesystem detects the physiological response (sinusitis). On the otherhand, FIG. 25b illustrates heating of the same ROI because of acne, forwhich the system does detect sinusitis. In one embodiment, the model wastrained on: samples generated from a first set of TH_(ROI1) andIM_(ROI2) taken while skin inflammation was detectable on a portion ofthe overlapping region between ROI₁ and ROI₂, and additional samplesgenerated from a second set of TH_(ROI1) and IM_(ROI2) taken while skininflammation was not detectable on the overlapping region. Optionally,utilizing a model trained so may enable the computer to account for skininflammation on the overlapping region.

Touching the ROI may affect TH_(ROI) by increasing or decreasing thetemperature at the touched region. Thus, touching the ROI may beconsidered a confounding factor that can make detections of thephysiological response less accurate. In one embodiment, the model wastrained on: samples generated from a first set of TH_(ROI1) andIM_(ROI2) taken while detecting that the user touches a portion of theoverlapping region between ROI₁ and ROI₂, and additional samplesgenerated from a second set of TH_(ROI1) and IM_(ROI2) taken whiledetecting that the user does not touch the overlapping region.Optionally, utilizing a model trained so may enables the computer toaccount for touching the overlapping region.

Throughout day-to-day activities, a user may make various facialmovements that are unrelated to the physiological response beingdetected, and thus can negatively affect the thermal measurements takenby CAM. This can lead to measurements that may be incorrectly attributedto the physiological response. To address this issue, the computer mayidentify disruptive activities, such as talking, eating, and drinking,and utilize the identified disruptive activities in order to moreaccurately detect the physiological response. In one embodiment, thecomputer utilizes a machine learning-based approach to handle thedisruptive activities. This approach may include (i) identifying, basedon IM_(ROI2), occurrences of one or more of the disruptive activities,(ii) generating feature values based on the identified disruptiveactivities, and (iii) utilizing a machine learning-based model to detectthe physiological response based on the feature values and featurevalues generated from TH_(ROI1).

In addition to detecting a physiological response, in some embodiments,the computer may utilize IM_(ROI2) to generate an avatar of the user(e.g., in order to represent the user in a virtual environment).Optionally, the avatar may express emotional responses of the user,which are detected based on IM_(ROI2). Optionally, the computer maymodify the avatar to show synthesized facial expressions that are notmanifested in the user's actual facial expressions, but the synthesizedfacial expressions correspond to emotional responses detected based onTH_(ROI1). Some of the various approaches that may be utilized togenerate the avatar based on IM_(ROI2) are described in co-pending USpatent publication 2016/0360970.

Contraction and relaxation of various facial muscles can cause facialtissue to slightly change its position and/or shape. Thus, facialmovements can involve certain movements to ROIs. With thermal camerasthat have multiple sensing elements (pixels), this can cause the ROI tomove and be covered by various subsets of pixels as the user's facemoves (e.g., due to talking/or making facial expressions). For example,smiling can cause the user's cheeks to move upwards. This can cause athermal camera that covers a cheek to capture an ROI located on a cheekwith a first set of pixels (from among the camera's pixels) when theuser has a neutral expression, and to capture images of the ROI with asecond set of pixels, when the user is smiling. In this example, onaverage, the pixels in the second set are likely to be located higher inthe images than the pixels in the first set. To account for the possiblemovement of ROIs due to facial expressions, the computer may tracklocations of one or more facial landmarks in a series of IM_(ROI2), andutilize the locations to adjust TH_(ROI1). Facial landmarks are usuallythe most salient facial points on the face.

In one embodiment in which CAM comprises multiple sensing elements,which correspond to values of multiple pixels in TH_(ROI1), the computermay assign weights to the multiple pixels based on the locations of theone or more facial landmarks, which are determined based on IM_(ROI2).Assigning weights to pixels based on their location with respect to afacial landmark can be considered a form of selection of the pixels thatcover the ROI based on the location of the landmark. In one example, theweights are assigned based on a function that takes into account thedistance of each pixel from the locations of one or more faciallandmarks and/or the relative position of each pixel with respect to thelocations.

In another embodiment, the computer may generate certain feature valuesbased on locations of one or more landmarks, which are determined basedon analysis of IM_(ROI2). These certain feature values may be utilizedin conjunction with other feature values (e.g., feature values generatedbased on TH_(ROI1)) to detect the physiological response using a machinelearning-based model.

The following is a description of a method for detecting a physiologicalresponse based on measurements from CAM and VCAM. The steps describedbelow may be performed by running a computer program having instructionsfor implementing the method. Optionally, the instructions may be storedon a computer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps: In Step 1, taking thermal measurements of a first ROI(TH_(ROI1)) on the user's face using an inward-facing head-mountedthermal camera located at most 15 cm from the user's face. In Step 2,taking images of a second ROI (IM_(ROI2)) on the user's face with aninward-facing head-mounted visible-light camera located at most 15 cmfrom the user's face. Optionally, the first ROI (ROI₁) and the secondROI (ROI₂) overlap. In Step 3, generating feature values based onTH_(ROI1) and IM_(ROI2). And in Step 4, utilizing a model to detect thephysiological response based on the feature values. Optionally, themodel was trained based on previous TH_(ROI1) and IM_(ROI2) taken ondifferent days.

In one embodiment, the physiological response is an emotional response,and the method optionally includes the following steps: calculating,based on IM_(ROI2), a value indicative of facial skin color changes(FSCC), and utilizing the value indicative of FSCC to generate at leastone of the feature values used to detect the physiological response inStep 4.

In another embodiment, generating the feature values in Step 3 involvesgenerating, based on IM_(ROI2), feature values indicative of anoccurrence of one or more of the following confounding factors on aportion of the overlapping region between ROI₁ and ROI₂: a presence ofcosmetics, a presence of sweat, a presence of hair, and a presence ofskin inflammation.

The following is a description of a system that detects a physiologicalresponse based on an inward-facing head-mounted thermal camera(CAM_(in)), an outward-facing head-mounted thermal camera (CAM_(out)),and a computer. CAM_(out) measures the environment and generates dataindicative of confounding factors, such as direct sunlight or airconditioning. Accounting for confounding factors enables the system tomore accurately detect the physiological response compared to a systemthat does not account for these confounding factors. Optionally,CAM_(in) and/or CAM_(out) are physically coupled to a frame worn on auser's head, such as a frame of a pair of eyeglasses or an augmentedreality device. Optionally, each of CAM_(in) and CAM_(out) weighs below5 g and is located less than 15 cm from the user's face.

CAM_(in) takes thermal measurements of an ROI (TH_(ROI)) on the user'sface. Optionally, CAM_(in) does not occlude the ROI. In one example, theROI includes a region on the forehead and the physiological responseinvolves stress, a headache, and/or a stroke. In another example, theROI includes a region on the nose and the physiological response is anallergic reaction.

CAM_(out) takes thermal measurements of the environment (TH_(ENV)).Optionally, CAM_(out) does not occlude the ROI. Optionally, the anglebetween the optical axes of CAM_(in) and CAM_(out) is at least 45°, 90°,130°, 170°, or 180°. Optionally, the field of view (FOV) of CAM_(in) islarger than the FOV of CAM_(out) and/or the noise equivalentdifferential temperature (NEDT) of CAM_(in) is lower than NEDT ofCAM_(out). In one example, CAM_(in) has a FOV smaller than 80° andCAM_(out) has a FOV larger than 80°. In another example, CAM_(in) hasmore sensing elements than CAM_(out) (e.g., CAM_(in) has at least doublethe number of pixels as CAM_(out)).

In one embodiment, CAM_(in) and CAM_(out) are based on sensors of thesame type with similar operating parameters. Optionally, CAM_(in) andCAM_(out) are located less than 5 cm or 1 cm apart. Having sensors ofthe same type, which are located near each other, may have an advantageof having both CAM_(in) and CAM_(out) be subject to similar inaccuraciesresulting from heat conductance and package temperature. In anotherembodiment, CAM_(in) and CAM_(out) may be based on sensors of differenttypes, with different operating parameters. For example, CAM_(in) may bebased on a microbolometer FPA while CAM_(out) may be based on athermopile (that may be significantly less expensive than themicrobolometer).

FIG. 28a illustrates one embodiment of the system that includesinward-facing and outward-facing head-mounted thermal cameras on bothsides of the frame. In this illustration, CAM_(in) is the inward-facingthermal camera 12, which takes thermal measurements of ROI 13, andCAM_(out) is the outward-facing thermal camera 62. Arc 64 illustratesthe larger FOV of CAM_(out) 62, compared to the FOV of CAM_(in) thatcovers ROI 13. The illustrated embodiment includes a second head-mountedthermal camera 10 (CAM_(in2)) on the right side of the frame, whichtakes thermal measurements of ROI 11, and a second outward-facinghead-mounted thermal camera 63 (CAM_(out2)). FIG. 28b illustratesreceiving an indication on a GUI (on the illustrated laptop) that theuser is not monitored in direct sunlight. Cameras 520 and 521 are theoutward-facing head-mounted thermal cameras.

The computer detects a physiological response based on TH_(ROI) andTH_(ENV). Optionally, TH_(ENV) are utilized to account for at least someof the effect of heat transferred from the environment to the ROI (andnot due to the user's physiological response). Thus, on average,detections of the physiological response based on TH_(ROI) and TH_(ENV)may be more accurate than detections of the physiological response basedon TH_(ROI) without TH_(ENV).

There are various ways in which the computer may utilize TH_(ENV) toincrease the accuracy of detecting the physiological response. In oneembodiment, the computer generates feature values based on a set ofTH_(ROI) and TH_(ENV), and utilizes a machine learning-based model todetect, based on the feature values, the physiological response. Byutilizing TH_(ENV) to generate one or more of the feature values, thecomputer may make different detections of the physiological responsebased on similar TH_(ROI) that are taken in dissimilar environments. Forexample, responsive to receiving a first set of measurements in whichTH_(ROI) reaches a first threshold while TH_(ENV) does not reach asecond threshold, the computer detects the physiological response.However, responsive to receiving a second set of measurements in whichTH_(ROI) reaches the first threshold while TH_(ENV) reaches the secondthreshold, the computer does not detect the physiological response.Optionally, TH_(ENV) reaching the second threshold indicates that theuser was exposed to high infrared radiation that is expected tointerfere with the detection.

In another embodiment, the computer may utilize TH_(ENV) for theselection of values that are appropriate for the detection of thephysiological response. In one example, the computer may selectdifferent thresholds (to which TH_(ROI) are compared) for detecting thephysiological response. In this example, different TH_(ENV) may causethe computer to use different thresholds. In another example, thecomputer may utilize TH_(ENV) to select an appropriate reference timeseries (to which TH_(ROI) may be compared) for detecting thephysiological response. In yet another example, the computer may utilizeTH_(ENV) to select an appropriate model to utilize to detect thephysiological response based on the feature values generated based onTH_(ROI).

In still another embodiment, the computer may normalize TH_(ROI) basedon TH_(ENV). In one example, the normalization may involve subtracting avalue proportional to TH_(ENV) from TH_(ROI), such that the value of thetemperature at the ROI is adjusted based on the temperature of theenvironment at that time and/or in temporal proximity to that time(e.g., using an average of the environment temperature during thepreceding minute). Additionally or alternatively, the computer mayadjust weights associated with at least some TH_(ROI) based on TH_(ENV),such that the weight of measurements from among TH_(ROI) that were takenduring times the measurements of the environment indicated extremeenvironmental temperatures is reduced.

In yet another embodiment, responsive to determining that TH_(ENV)represent an extreme temperature (e.g., lower than 5° C., higher than35° C., or some other ranges deemed inappropriate temperatures), thecomputer may refrain from performing detection of the physiologicalresponse. This way, the computer can avoid making a prediction that isat high risk of being wrong due to the influence of the extremeenvironmental temperatures. In a similar manner, instead of determiningthat TH_(ENV) represent an extreme temperature, the computer maydetermine that the difference between TH_(ROI) and TH_(ENV) are not inan acceptable range (e.g., there is a difference of more than 15° C.between the two), and refrain from making a detection of thephysiological response in that event.

The following examples describe ways to use TH_(ENV) to detect thephysiological response based on TH_(ROI). In one example, the computerdetects the physiological response based on a difference betweenTH_(ROI) and TH_(ENV), which enables the system to operate well in anuncontrolled environment that does not maintain environmentaltemperature in a range below ±1° C. and does not maintain humidity in arange below ±3%. In another example, the computer detects thephysiological response by performing the following steps: calculating atemperature difference between TH_(ROI) and TH_(ENV) taken at time x(ΔT_(x)), calculating a temperature difference between TH_(ROI) andTH_(ENV) taken at time y (ΔT_(y)), and detecting the physiologicalresponse based on a difference between ΔT_(x) and ΔT_(y). Optionally,detecting the physiological response is based on the difference betweenΔT_(x) and ΔT_(y) reaching a predetermined threshold. Optionally, thepredetermined threshold is selected from a threshold in the time domain,and/or a threshold in the frequency domain. Optionally, the magnitude ofthe difference between ΔT_(x) and ΔT_(y) is indicative of an extent ofthe physiological response. It is noted that sentences such as“calculating a difference between M and N” or “detecting a differencebetween M and N” are intended to cover any function that is proportionalto the difference between M and N.

Because the FOV of CAM_(out) is limited and the responsivity ofCAM_(out) decreases when drawing away from the optical axis, it may bebeneficial to utilize two or more CAM_(out) pointed at different angles.

In one embodiment, the system may include a second outward-facinghead-mounted thermal camera (CAM_(out2)), which takes thermalmeasurements of the environment (TH_(ENV2)). Optionally, there is anangle of at least 30° between the optical axes of CAM_(out) andCAM_(out2). Utilizing two or more outward-facing head-mounted thermalcameras such as CAM_(out) and CAM_(out2) can help identify cases inwhich there is a directional environmental interference (e.g., sunlightcoming from a certain direction). In some cases, such a directionalinterference can lead to refraining from making a detection of thephysiological response. For example, responsive to receiving a first setof measurements in which TH_(ROI) reach a first threshold while thedifference between TH_(ENV) and TH_(ENV2) does not reach a secondthreshold, the computer detects the physiological response. However,responsive to receiving a second set of measurements in which TH_(ROI)reach the first threshold while the difference between TH_(ENV) andTH_(ENV2) reaches the second threshold, the computer does not detect thephysiological response. Optionally, the computer detects thephysiological response based on a difference between TH_(ROI), TH_(ENV),and TH_(ENV2), while taking into account the angle between the opticalaxes of CAM_(out) and CAM_(out2) and a graph of responsivity as functionof the angle from the optical axes of each of CAM_(out) and CAM_(out2).

In another embodiment, CAM_(in) and CAM_(out) are located to the rightof the vertical symmetry axis that divides the user's face, and the ROIis on the right side of the face. Optionally, the system includes asecond inward-facing head-mounted thermal camera (CAM_(in2)) and asecond outward-facing head-mounted thermal camera (CAM_(out2)) locatedto the left of the vertical symmetry axis. CAM_(in2) takes thermalmeasurements of a second ROI (TH_(ROI2)) on the left side of the face,and does not occlude the second ROI (ROI₂). CAM_(out2) takes thermalmeasurements of the environment (TH_(ENV2)) that is more to the leftrelative to TH_(ENV). In this embodiment, the computer detects thephysiological response also based on TH_(ROI2) and TH_(ENV2).

In still another embodiment, the optical axes of CAM_(in) and CAM_(out)are above the Frankfort horizontal plane, and the system furtherincludes a second inward-facing head-mounted thermal camera (CAM_(in2))and a second outward-facing head-mounted thermal camera (CAM_(out2)),located such that their optical axes are below the Frankfort horizontalplane, which take thermal measurements TH_(ROI2) and TH_(ENV2),respectively. In this embodiment, the computer detects the physiologicalresponse also based on TH_(ROI2) and TH_(ENV2).

Optionally, the computer detects the physiological response byperforming at least one of the following calculations: (i) when thedifference between TH_(ENV) and TH_(ENV2) reaches a threshold, thecomputer normalizes TH_(ROI) and TH_(ROI2) differently against thermalinterference from the environment, (ii) when TH_(ENV) does not reach apredetermined threshold for thermal environmental interference, whileTH_(ENV2) reaches the predetermined threshold, the computer assignsTH_(ROI) a higher weight than TH_(ROI2) for detecting the physiologicalresponse, and (iii) the computer generates feature values based onTH_(ROI), TH_(ENV), TH_(ENV2) and optionally TH_(ROI2) and utilizes amodel to detect, based on the feature values, the physiologicalresponse. Optionally, the model was trained based on a first set ofTH_(ROI), TH_(ROI2), TH_(ENV) and TH_(ENV2) of one or more users takenwhile the one or more users had the physiological response, and a secondset of TH_(ROI), TH_(ROI1), TH_(ENV) and TH_(ENV2) of the one or moreusers taken while the one or more users did not have the physiologicalresponse.

In addition to having one or more CAM_(out), or instead of having theone or more CAM_(out), some embodiments may include a sensor that may beused to address various other confounding factors, such as usermovements and wind, which are discussed below. Optionally, the sensor iscoupled to a frame worn on the user's head. An example of such a sensoris sensor 68 in FIG. 28 a.

In one embodiment, the sensor takes measurements (denoted m_(conf)) thatare indicative of an extent of the user's activity, an orientation ofthe user's head, and/or a change in a position of the user's body. Forexample, the sensor may be (i) a movement sensor that is physicallycoupled to a frame worn on the user's head, or coupled to a wearabledevice worn by the user, (ii) a visible-light camera that takes imagesof the user, and/or (iii) an active 3D tracking device that emitselectromagnetic waves and generates 3D images based on receivedreflections of the emitted electromagnetic waves. Optionally, thecomputer detects the physiological response also based on m_(conf). Inone example, the computer may refrain from detecting the physiologicalresponse if m_(conf) reaches a threshold (which may indicate the userwas very active which causes an increase in body temperature). Inanother example, the computer generates feature values based onTH_(ROI), TH_(ENV), and m_(conf) and utilizes a model to detect thephysiological response based on the feature values. Optionally, themodel was trained based on previous TH_(ROI), TH_(ENV), and m_(conf)taken while the user had different activity levels. For example, themodel may be trained based on: a first set of previous TH_(ROI),TH_(ENV), and m_(conf) taken while the user was walking or running, anda second set of previous TH_(ROI), TH_(ENV), and m_(conf) taken whilethe user was sitting or standing.

FIG. 30 illustrates an elderly person whose facial temperature increasesas a result of bending the head down towards the floor. In this example,the system receives an indication of the user's action via the sensor(e.g., one or more gyroscopes) and consequently refrains fromerroneously detecting certain physiological responses, since theincrease in temperature may be attributed to the person being bent over.In one embodiment, a sensor provides indications indicative of bendingthe head down above a certain degree from the normal to earth, wherebending the head down above the certain degree is expected to cause achange in TH_(ROI). The computer generates feature values based onTH_(ROI), TH_(ENV), and m_(conf), and utilizes a model to detect thephysiological response based on the feature values. The model wastrained based on: a first set of previous TH_(ROI), TH_(ENV), andm_(conf) taken while a user was bending the head down above the certaindegree, and a second set of previous TH_(ROI), TH_(ENV), and m_(conf)taken while the user was not bending the head down above the certaindegree.

In another embodiment, the sensor may be an anemometer that isphysically coupled to a frame worn on the user's head, is located lessthan 15 cm from the face, and provides a value indicative of a speed ofair directed at the face (m_(wind)). Optionally, the computer detectsthe physiological response also based on m_(wind). In one example, thecomputer refrains from detecting the physiological response if m_(wind)reaches a threshold (which may indicate that the user was in anenvironment with strong wind that may excessively cool regions on theface). In another example, the computer generates feature values basedon TH_(ROI), TH_(ENV), and m_(wind) and utilizes a model to detect,based on the feature values, the physiological response. FIG. 29illustrates a case in which a user receives an indication that she isnot being monitored in a windy environment. Optionally, the model wastrained based on previous TH_(ROI), TH_(ENV), and m_(wind) taken while auser was in different environments. For example, the model may betrained based on: a first set of previous TH_(ROI), TH_(ENV), andm_(wind) taken while being indoors, and a second set of previousTH_(ROI), TH_(ENV), and m_(wind) taken while being outdoors.

The following is a method for detecting a physiological response whiletaking into account a confounding factor that involves environmentalthermal interferences (e.g., direct sunlight). Having differentenvironmental conditions may cause a system such as the one illustratedin FIG. 28a to behave differently, as shown in the steps below. Thesteps described below may be performed by running a computer programhaving instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps: In Step 1, takingthermal measurements of a region of interest (TH_(ROI)) on a user's faceutilizing an inward-facing head-mounted thermal camera (CAM_(in)) wornby the user. In step 2, taking thermal measurements of the environment(TH_(ENV)) utilizing an outward-facing head-mounted thermal camera(CAM_(out)) worn by the user. In step 3, generating feature values basedon TH_(ROI) and TH_(ENV). And in step 4, utilizing a machinelearning-based model to detect the physiological response based on thefeature values.

The method may optionally further include the following steps: taking afirst set of TH_(ROI) (first TH_(ROI)), where the first set of TH_(ROI)reach a first threshold; taking a first set of TH_(ENV) (firstTH_(ENV)), where the first set of TH_(ENV) do not reach a secondthreshold; detecting, based on the first set of TH_(ROI) and the firstset of TH_(ENV), that the user had the physiological response; taking asecond set of TH_(ROI), where the second set of TH_(ROI) reach the firstthreshold; taking a second set of TH_(ENV), where the second set ofTH_(ENV) reach the second threshold; and detecting, based on the secondset of TH_(ROI) and the second set of TH_(ENV), that the user did nothave the physiological response. Optionally, the method furtherincludes: taking a third set of TH_(ROI), where the third set ofTH_(ROI) do not reach the first threshold; taking a third set ofTH_(ENV), where the third set of TH_(ENV) do not reach the secondthreshold; and detecting, based on the third set of TH_(ROI) and thethird set of TH_(ENV), that the user did not have the physiologicalresponse.

The following is a description of a system for detecting a physiologicalresponse, which includes a CAM and a sensor. The sensor providesmeasurements indicative of times at which the user touches the face.Touching the face can warm certain regions of the face, and the systemmay utilize these measurements in order to account for such cases. Thus,the system may more accurately detect the physiological responsecompared to systems that do not account for touching of the face.

CAM is worn on the user's head and takes thermal measurements of an ROI(TH_(ROI)) on the user's face. Optionally, the system includes a frameto which CAM and the sensor may be physically coupled. Optionally, CAMis located less than 15 cm from the face and/or weighs below 10 g.

The sensor provides measurements (M) indicative of times at which theuser touches the ROI. The user may touch the ROI using/with a finger,the palm, a tissue or a towel held by the user, a makeup-related itemheld by the user, and/or a food item eaten by the user. Touching the ROImay affect TH_(ROI) by increasing or decreasing the temperature at thetouched region. Thus, touching the ROI may be considered a confoundingfactor that can make detections of the physiological response by acomputer less accurate. M may include values measured by the sensorand/or results of processing of values measured by the sensor. Varioustypes of sensors may be utilized in different embodiments to generate M,such as: a visible-light camera (where the computer uses imageprocessing to identify touching the ROI), a miniature radar (such aslow-power radar operating in the range between 30 GHz and 3,000 GHz,where the computer uses signal processing of the reflections to identifytouching the ROI), a miniature active electro-optics distancemeasurement device, and/or an ultrasound sensor.

In some embodiments, the sensor may be unattached to a frame worn on theuser's head. For example, the sensor may include a visible-light cameramounted to an object in the user's environment (e.g., a laptop), and maynormally located at a distance greater than 20 cm from the user's face.Optionally, the computer may utilize M to determine when it is likely(but not necessarily certain) that the user touched the face. In oneexample, the sensor includes a movement-measuring device embedded in abracelet, and the computer increases the probability for a physicalcontact with the face when the user's hand is estimated to be at facelevel and/or close to the face. In another example, the sensor includesan altimeter embedded in a bracelet, and the computer increases theprobability for an event of physical contact with the face when theuser's hand is estimated to be at face level.

FIG. 26a and FIG. 26b illustrate one embodiment of a system thatprovides indications when the user touches his/her face. The systemincludes a frame 533, head-mounted sensors (530, 531, 532) able todetect touching the face, and head-mounted thermal cameras (534, 535,536, 537). Optionally, the head-mounted sensors are visible-lightcameras that take images of the ROIs. Head-mounted sensor 530 capturesan ROI above the frame, and head-mounted sensors 531 and 532 captureROIs below the frame. Hot spot 538, which is measured by the thermalcamera 534, was caused by touching the forehead and is unrelated to thephysiological response being detected. Upon detecting touching of theROI, the computer may use the associated thermal measurementsdifferently than it would use had the touching not been detected.Additionally or alternatively, a user interface may provide anindication that touching the ROI hinders the detection of thephysiological response.

The computer detects the physiological response based on TH_(ROI) and M.Optionally, since the computer utilizes M to account, at least in part,for the effect of touching the face, on average, detections of thephysiological response based on TH_(ROI) and M are more accurate thandetections of the physiological response based on TH_(ROI) without M.The computer may utilize TH_(ROI) in various ways in order to detect thephysiological response, such as comparing one or more values derivedfrom TH_(ROI) to a threshold and/or comparing TH_(ROI) to a referencetime series.

Another approach that may be utilized involves a machine learning-basedmodel. In one embodiment, the computer generates feature values based onTH_(ROI) and M, and utilizes the model to detect, based on the featurevalues, the physiological response. By utilizing M to generate one ormore of the feature values, the computer may make different detectionsof the physiological response based on similar TH_(ROI) that are takenwhile there are different extents of touching the ROI. For example,responsive to receiving a first set of measurements in which TH_(ROI)reaches a threshold, while M indicate that there was no touching of theROI, the computer detects the physiological response. However,responsive to receiving a second set of measurements in which TH_(ROI)reaches the threshold, but M indicate that the user touched the ROI, thecomputer does not detect the physiological response. Optionally, themodel is trained based on samples, each comprising: (i) feature valuesgenerated based on TH_(ROI) taken while M indicates touching the ROI,and (ii) a corresponding label indicative of an extent of thephysiological response. Optionally, the samples include: a first set ofsamples with labels corresponding to having the physiological response,which are generated based on M indicating that the ROI was not touched,and a second set of samples with labels corresponding to not having thephysiological response, which are generated based on M indicating thatthe ROI was touched. Optionally, the samples comprise: a third set ofsamples with labels corresponding to having the physiological response,which are generated based on M indicating that the ROI was touched,and/or a fourth set of samples with labels corresponding to not havingthe physiological response, which are generated based on M indicatingthat the ROI was not touched.

M may be utilized by the computer in order to decrease the chance ofmaking incorrect detections of the physiological response. In oneembodiment, the computer utilizes, for the detection of thephysiological response, TH_(ROI) taken at times in which M are notindicative of touching the ROI. In this embodiment, the computer doesnot utilize, for the detection of the physiological response, TH_(ROI)taken at times in which M are indicative of touching the ROI. In anotherembodiment, the computer does not utilize, for the detection of thephysiological response, TH_(ROI) taken during at least one of thefollowing intervals starting after M indicate that the user touched theROI: 0-10 seconds, 0-30 second, 0-60 second, 0-180 seconds, and 0-300seconds. In yet another embodiment, the computer attributes, for thedetection of the physiological response, a smaller weight to TH_(ROI)taken during a certain interval starting after M indicate that the usertouched the ROI, compared to a weight attributed to TH_(ROI) taken attimes shortly before M indicate that the user touched the ROI.Optionally, the certain interval includes at least one of the followingdurations: 10-30 second, 30-60 second, 60-120 seconds, and 120-300seconds. Optionally, the higher the weight attributed to a measurement,the more it influences calculations involved in the detection of thephysiological response.

In one embodiment, the system optionally includes a user interface (UI)which notifies the user about touching the ROI. Optionally, thisnotification is in lieu of notifying extent of the physiologicalresponse corresponding to the time the user touched the ROI. Thenotification may be delivered to the user using a sound, a visualindication on a head-mounted display, and/or a haptic feedback.Optionally, the UI includes a screen of an HMS (e.g., a screen of anaugmented reality headset), a screen of a device carried by the user(e.g., a screen of a smartphone or a smartwatch), and/or a speaker(e.g., an earbud or headphones). Optionally, the computer identifiesthat the duration and/or extent of touching the face reached athreshold, and then commands the UI to alert the user that an accuratedetection of the physiological response cannot be made as long as thetouching continues.

In one embodiment, the sensor includes a visible-light camera and/or anear-infrared camera, the system is powered by a battery, and the systemmay operate in a state belonging to a set comprising first and secondstates. While operating in the first state, the system checks on aregular basis whether the user touches the ROI. While operating in thesecond state, the system checks whether the user touches the ROI inresponse to detecting abnormal TH_(ROI). Optionally, the system consumesless power while operating in the second state compared to the power itconsumes while operating in the first state.

In one embodiment, the measurements taken by the sensor are furtherindicative of an angular position of CAM relative to the ROI while theframe is still worn on the head, and the computer detects thephysiological response also based on the angular position. Optionally,the measurements of the angular position are utilized to account forinstances in which the frame has moved, and consequently CAM captures aregion that only overlaps, or does not overlap at all, with the intendedROI. Optionally, the computer is able to detect changes below 5° in theangular position, which may also influence TH_(ROI). Thus, on average,detections of the physiological response based on TH_(ROI) and theangular position are more accurate compared to detections of thephysiological responses based on TH_(ROI) without the angular position,while the frame is still worn on the head.

In a first example, responsive to the angular position of CAM relativeto the ROI reaching a predetermined threshold, the computer refrainsfrom detecting the physiological response and/or alerts the user.

In a second example, the computer generates feature values based onTH_(ROI) and the angular position, and utilizes a model to detect thephysiological response based on the feature values. Optionally, themodel was trained based on data comprising TH_(ROI) collected while CAMwas at different distances and/or angular positions relative to the ROI.Thus, the model may account, in its parameters, for various effects thatthe distance and/or orientation of CAM may have on TH_(RO) in order tomore accurately detect the physiological response.

In a third example, the sensor includes a visible-light camera thattakes images of a region on the user's face, and the computer calculatesthe angular position of the visible-light camera relative to the facebased on analyzing the images, and then calculates the angular positionof CAM relative to the ROI based on a predetermined transformationbetween the angular position of the visible-light camera relative to theface and the angular position of CAM relative to the ROI.

In a fourth example, the sensor includes a transceiver ofelectromagnetic waves, and the computer calculates the angular positionof the transceiver relative to the face based on signal processing ofthe reflections from the face, and then calculates the angular positionof CAM relative to the ROI based on a predetermined transformationbetween the angular position of the transceiver relative to the face andthe angular position of CAM relative to the ROI.

The following method for detecting a physiological response may be used,in some embodiments, by the system described above, which detects aphysiological response while taking into account a confounding factorsuch as touching the face. The steps described below may be performed byrunning a computer program having instructions for implementing themethod. Optionally, the instructions may be stored on acomputer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps: In Step 1, taking thermal measurements of an ROI(TH_(ROI)) on a user's face using an inward-facing head-mounted thermalcamera. In Step 2, taking, utilizing a sensor, measurements (M)indicative of times at which the user touches the ROI. Touching the ROImay affect TH_(ROI), for example by increasing the temperatures at theROI (which may increase the values of TH_(ROI)). The sensor may be ahead-mounted sensor or a sensor that is not head-mounted. And in Step 3,detecting the physiological response based on TH_(ROI) and M. Forexample, the detection may be performed by the computer, as describedabove. On average, detections of the physiological response based onTH_(ROI) and M are more accurate compared to detections of thephysiological response based on TH_(ROI) without M.

Optionally, the method further includes the following steps: generatingfeature values based on TH_(ROI) and M, and utilizing a model fordetecting the physiological response based on the feature values.Optionally, the model was trained based on samples, each comprising: (i)feature values generated based on previous TH_(ROI) taken while Mindicates touching the ROI, and (ii) a corresponding label indicative ofan extent of the physiological response. Optionally, the samplesinclude: a first set of samples with labels corresponding to having thephysiological response, which are generated based on M indicating thatthe ROI was not touched, and a second set of samples with labelscorresponding to not having the physiological response, which aregenerated based on M indicating that the ROI was touched.

Optionally, M are further indicative of angular position of CAM relativeto the ROI, while the frame is still worn on the head. And the methodfurther includes a step of detecting the physiological response alsobased on the angular position. On average, detections of thephysiological response based on TH_(ROI) and the angular position aremore accurate compared to detections of the physiological responsesbased on TH_(ROI) without the angular position, while the frame is stillworn on the head.

The following is a description of a system that detects a physiologicalresponse while taking into account a consumption of a confoundingsubstance. When a person consumes a confounding substance, it may affectthermal measurements of an ROI (TH_(ROI)) on the person's face. Theaffect to TH_(ROI) can be attributed to various physiological and/ormetabolic processes that may ensue following the consumption of theconfounding substance, which can result (amongst possibly other effects)in a raising or decreasing of the temperature at the ROI in a mannerthat is unrelated to the physiological response being detected. Thus,embodiments of this system utilize indications indicative of consumptionof a confounding substance (such as medication, an alcoholic beverage, acaffeinated beverage, and/or a cigarette) to improve the system'sdetection accuracy. In one embodiment, the system includes a CAM and acomputer.

CAM is worn on the user's head and takes thermal measurements of an ROI(TH_(ROI)) on the user's face. Optionally, the system includes a frameto which CAM and the device are physically coupled. Optionally, CAM islocated less than 15 cm from the face and/or weighs below 10 g.

In different embodiments, the ROI may cover different regions on theface and CAM may be located at different locations on a frame worn onthe user's head and/or at different distances from the user's face. Inone embodiment, the ROI is on the forehead, and CAM is physicallycoupled to an eyeglasses frame, located below the ROI, and does notocclude the ROI. Optionally, the physiological response detected in thisembodiment is stress, a headache, and/or a stroke. In anotherembodiment, the ROI is on the periorbital area, and CAM is located lessthan 10 cm from the ROI. Optionally, the physiological response detectedin this embodiment is stress. In yet another embodiment, the ROI is onthe nose, and CAM is physically coupled to an eyeglasses frame and islocated less than 10 cm from the face. Optionally, the physiologicalresponse detected in this embodiment is an allergic reaction. In stillanother embodiment, the ROI is below the nostrils, and CAM: isphysically coupled to an eyeglasses frame, located above the ROI, anddoes not occlude the ROI. Optionally, the ROI covers one or more areason the upper lip, the mouth, and/or air volume(s) through which theexhale streams from the nose and/or mouth flow, and the physiologicalresponse detected in this embodiment is a respiratory parameter such asthe user's breathing rate.

The computer may receive, from a device, an indication indicative ofconsuming a confounding substance that is expected to affects TH_(ROI),such as an alcoholic beverage, a medication, caffeine, and/or acigarette. Various types of devices may be utilized in differentembodiments in order to identify consumption of various confoundingsubstances.

In one embodiment, the device includes a visible-light camera that takesimages of the user and/or the user's environment. Optionally, thevisible-light camera is a head-mounted visible-light camera having inits field of view a volume that protrudes out of the user's mouth.Optionally, the computer identifies a consumption of the confoundingsubstance based on analyzing the images. In one example, thevisible-light camera may belong to a camera-based system such as OrCam(http://www.orcam.com/), which is utilized to identify various objects,products, faces, and/or recognize text. In another example, imagescaptured by the visible-light camera may be utilized to determine thenutritional composition of food a user consumes. Such an approach inwhich images of meals are utilized to generate estimates of food intakeand meal composition, is described in Noronha, et al., “Platemate:crowdsourcing nutritional analysis from food photographs”, Proceedingsof the 24th annual ACM symposium on User interface software andtechnology, ACM, 2011. Additional examples of how a visible-light cameramay be utilized to identify consumption of various substances is givenin U.S. Pat. No. 9,053,483 (Personal audio/visual system providingallergy awareness) and in U.S. Pat. No. 9,189,021 (Wearable foodnutrition feedback system).

In another embodiment, the device includes a microphone that records theuser, and the computer identifies a consumption of the confoundingsubstance utilizing a sound recognition algorithm operated on arecording of the user. Optionally, the sound recognition algorithmcomprises a speech recognition algorithm configured to identify wordsthat are indicative of consuming the confounding substance.

In yet another embodiment, the confounding substance is a medication,and the device includes a pill dispenser that provides an indicationindicating that the user took a medication, and/or a sensor-enabled pillthat includes an ingestible signal generator coupled to a medicationthat is configured to generate a body-transmissible signal uponingestion by a user, such as the sensor-enabled pill described in PCTpublication WO/2016/129286. Optionally, the indication indicates thetype of medication and/or its dosage.

In still another embodiment, the device is a refrigerator, a pantry,and/or a serving robot. Optionally, the device provides an indicationindicative of the user taking an alcoholic beverage and/or a food item.

In yet another embodiment, the device has an internet-of-things (IoT)capability through which the indication is provided to the system. Forexample, the device may be part of a “smart device” with networkconnectivity.

And in yet another embodiment, the device belongs to a user interfacethat receives an indication from the user or/or a third party about theconsuming of the confounding substance.

Due to various metabolic and/or other physiological processes,consumption of a confounding substance may affect TH_(ROI). For example,many drugs are known to act on the hypothalamus and other brain centersinvolved in controlling the body's thermoregulatory system. Herein,stating “the confounding substance affects TH_(ROI)” means thatconsuming the confounding substance leads to a measurable change of thetemperature at the ROI, which would likely not have occurred had theconfounding substance not been consumed. Similarly, a time in which“confounding substance did not affect TH_(ROI)” is a time that occursafter at least a certain duration has elapsed since the confoundingsubstance was last consumed (or was not consumed at all), and theconsumption of the confounding substance is no longer expected to have anoticeable effect on the ROI temperature. This certain duration maydepend on factors such as the type of substance, the amount consumed,and previous consumption patterns. For example, the certain duration maybe at least: 30 minutes, two hours, or a day.

The duration of the effect of a confounding substance may vary betweensubstances, and may depend on various factors such as the amount ofsubstance, sex, weight, genetic characteristics, and the user's state.For example, consumption of alcohol on an empty stomach often has agreater effect on TH_(ROI) than consumption of alcohol with a meal. Someconfounding substances may have a long-lasting effect, possiblythroughout the period they are taken. For example, hormonalcontraceptives can significantly alter daily body temperatures. Otherconfounding factors, such as caffeine and nicotine, may have shorterlasting effects, typically subsiding within less than an hour or twofollowing their consumption.

The computer detects the physiological response based on TH_(ROI) andthe indication indicative of consuming the confounding substance. In oneembodiment, the computer refrains from detecting the physiologicalresponse within a certain window during which the confounding substanceaffected the user (e.g., an hour, two hours, or four hours). In anotherembodiment, the computer utilizes a model, in addition to TH_(ROI) andthe indication, to detect whether the user had the physiologicalresponse during the time that a consumed confounding substance affectedTH_(ROI). Optionally, the computer detects the physiological response bygenerating feature values based on TH_(ROI) and the indication (andpossibly other sources of data), and utilizing the model to calculate,based on the feature values, a value indicative of the extent of thephysiological response. Optionally, the feature values include a featurevalue indicative of one or more of the following: the amount of theconsumed confounding substance, the dosage of the consumed confoundingsubstance, the time that has elapsed since the confounding substance hadlast been consumed, and/or the duration during which the confoundingfactor has been consumed (e.g., how long the user has been taking acertain medication).

In one embodiment, the model was trained based on data collected fromthe user and/or other users, which includes TH_(ROI), the indicationsdescribed above, and values representing the physiological responsecorresponding to when TH_(ROI) were taken. Optionally, the data is usedto generate samples, with each sample comprising feature values and alabel. The feature values of each sample are generated based on TH_(ROI)taken during a certain period and an indication indicating whether aconfounding substance affected TH_(ROI) taken during the certain period.The label of the sample is generated based on one or more of the valuesrepresenting the physiological response, and indicates whether (andoptionally to what extent) the measured user had the physiologicalresponse during the certain period. Optionally, the data used to trainthe model reflects both being affected and being unaffected by theconfounding substance. For example, the data used to train the model mayinclude: a first set of TH_(ROI) taken while the confounding substanceaffected TH_(ROI), and a second set of TH_(ROI) taken while theconfounding substance did not affect TH_(ROI). Optionally, each of thefirst and second sets comprises at least some TH_(ROI) taken while themeasured user had the physiological response and at least some TH_(ROI)taken while the measured user did not have the physiological response.

Using the indications (indicative of the user consuming a confoundingsubstance) may lead to cases where the detection of the physiologicalresponse depends on whether the confounding substance was consumed. Inone example, in which the physiological response is detected whenTH_(ROI) reach a threshold, the computer's detection behavior may be asfollows: the computer detects the physiological response based on firstTH_(ROI) for which there is no indication indicating that the firstTH_(ROI) were affected by a consumption of the confounding substance,and the first TH_(ROI) reach the threshold; the computer does not detectthe physiological response based on second TH_(ROI) for which there isan indication indicating that the second TH_(ROI) were affected by aconsumption of the confounding substance, and the second TH_(ROI) alsoreach the threshold; and the computer does not detect the physiologicalresponse based on third TH_(ROI) for which there is no indicationindicating that the third TH_(ROI) were affected by a consumption theconfounding substance, and the third TH_(ROI) do not reach thethreshold.

The following three figures illustrate scenarios where issuing of alertsare dependent on the consumption of confounding substances. FIG. 31illustrates that the effect of consuming alcohol on a certain TH_(ROI)usually decreases after duration typical to the user (e.g., the durationis based on previous measurements of the user). Thus, when the effectremains high there may be a problem and the system may issue an alert.The figure illustrates an outward-facing visible-light camera 525 thatgenerates the indications indicative of when the user consumes alcoholicbeverages.

FIG. 32 illustrates a usual increase in a certain TH_(ROI) while theuser smokes. The system identifies when the user smoked (e.g., based onimages taken by the outward-facing visible-light camera 525) and doesn'talert because of an increase in TH_(ROI) caused by the smoking. However,when the temperature rises without the user having smoked for a certaintime, then it may be a sign that there is a problem, and the user mightneed to be alerted.

FIG. 33 illustrates the expected decrease in a certain TH_(ROI) afterthe user takes medication, based on previous TH_(ROI) of the user. Thesystem identifies when the medication is consumed, and does not generatean alert at those times. However, when TH_(ROI) falls without medicationhaving been taken, it may indicate a physiological response of which theuser should be made aware.

The following method for detecting a physiological response while takinginto account consumption of a confounding substance may be used, in someembodiments, by the system described above, which detects aphysiological response while taking into account a consumption of aconfounding substance. The steps described below may be performed byrunning a computer program having instructions for implementing themethod. Optionally, the instructions may be stored on acomputer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps:

In Step 1, taking thermal measurements of an ROI (TH_(ROI)) on theuser's face utilizing an inward-facing head-mounted thermal camera.

In Step 2, receiving an indication indicative of consuming a confoundingsubstance that affects TH_(ROI). Optionally, the indication is receivedfrom one or more of the various device described above that provide anindication indicative of consuming a confounding substance. Optionally,the indication is generated based on image processing of images taken bya head-mounted visible-light camera having in its field of a volume thatprotrudes out of the user's mouth.

And in Step 3, detecting the physiological response, while the consumedconfounding substance affects TH_(ROI), based on TH_(ROI), theindication, and a model. Optionally, the model was trained on: a firstset of TH_(ROI) taken while the confounding substance affected TH_(ROI),and a second set of TH_(ROI) taken while the confounding substance didnot affect TH_(ROI). Optionally, the model is a machine learning-basedmodel, and this step involves: generating feature values based onTH_(ROI) and the indication, and utilizing the machine learning-basedmodel to detect the physiological response based on the feature values.

One way in which a user may wear a head-mounted camera (such as CAM orVCAM) involves attaching a clip-on device that houses the camera onto aframe worn by the user, such as an eyeglasses frame. This may enable theuser to be selective regarding when to use the head-mounted camera andtake advantage of eyeglasses that he or she owns, which may becomfortable and/or esthetically pleasing.

In some embodiments, the clip-on device includes a body that may beattached and detached, multiple times, from a pair of eyeglasses inorder to secure and release the clip-on device from the eyeglasses. Thebody is a structure that has one or more components fixed to it. Forexample, the body may have one or more inward-facing camera fixed to it.Additionally, the body may have a wireless communication module fixed toit. Some additional components that may each be optionally fixed to thebody include a processor, a battery, and one or more outward-facingcameras.

In one example, “eyeglasses” are limited to prescription eyeglasses,prescription sunglasses, plano sunglasses, and/or augmented realityeyeglasses. This means that “eyeglasses” do not refer to helmets, hats,virtual reality devices, and goggles designed to be worn overeyeglasses. Additionally or alternatively, neither attaching the clip-ondevice to the eyeglasses nor detaching the clip-on device from theeyeglasses should take more than 10 seconds for an average user. Thismeans that manipulating the clip-on device is not a complicated task.Optionally, the body is configured to be detached from the eyeglasses bythe user who wears the eyeglasses, who is not a technician, and withoutusing a tool such as a screwdriver or a knife. Thus, the clip-on devicemay be attached and detached as needed, e.g., enabling the user toattach the clip-on when there is a need to take measurements, andotherwise have it detached.

In order to be warn comfortably, possibly for long durations, theclip-on device is a lightweight device, weighing less than 40 g (i.e.,the total weight of the body and the components fixed to it is less than40 g). Optionally, the clip-on device weighs below 20 g and/or below 10g.

The body is a structure to which components (e.g., an inward-facingcamera) may be fixed such that the various components do not fall offwhile the clip-on device is attached to the eyeglasses. Optionally, atleast some of the various components that are fixed to the body remainin the same location and/or orientation when the body is attached to theeyeglasses. Herein, stating that a component is “fixed” to the body isintended to indicate that, during normal use (e.g., involvingsecuring/releasing the clip-on device), the components are typically notdetached from the body. This is opposed to the body itself, which innormal use is separated from the eyeglasses frame, and as such, is notconsidered “fixed” to the eyeglasses frame.

In some embodiments, the body is a rigid structure made of a materialsuch as plastic, metal, and/or an alloy (e.g., carbon alloy).Optionally, the rigid structure is shaped such that it fits the contoursof at least a portion of the frame of the eyeglasses in order to enablea secure and stable attachment to the eyeglasses. In other embodiments,the body may be made of a flexible material, such as rubber. Optionally,the flexible body is shaped such that it fits the contours of at least aportion of the frame of the eyeglasses in order to enable a secure andstable attachment to the eyeglasses. Additionally or alternatively, theflexible body may assume the shape of a portion of the frame when it isattached to the eyeglasses.

The body may utilize various mechanisms in order to stay attached to theeyeglasses. In one embodiment, the body may include a clip memberconfigured to being clipped on the eyeglasses. In another embodiment,the body may include a magnet configured to attach to a magnet connectedto the eyeglasses and/or to a metallic portion of the eyeglasses. In yetanother embodiment, the body may include a resting tab configured tosecure the clip-on to the eyeglasses. In still another embodiment, thebody may include a retention member (e.g., a clasp, buckle, clamp,fastener, hook, or latch) configured to impermanently couple the clip-onto the eyeglasses. For example, clasp 147 is utilized to secure theclip-on device illustrated in FIG. 15a to the frame of the eyeglasses.And in yet another embodiment, the body may include a spring configuredto apply force that presses the body towards the eyeglasses. An exampleof this type of mechanism is illustrated in FIG. 17a where spring 175 isused to apply force that pushes body 170 and secures it in place toframe 176.

Herein, to “impermanently couple” something means to attach in a waythat is easily detached without excessive effort. For example, couplingsomething by clipping it on or closing a latch is consideredimpermanently coupling it. Coupling by screwing a screw with ascrewdriver, gluing, or welding is not considered impermanentlycoupling. The latter would be examples of what may be considered to“fix” a component to the body.

The inward-facing camera is fixed to the body. It takes images of aregion of interest on the face of a user who wears the eyeglasses.Optionally, the inward-facing camera remains pointed at the region ofinterest even when the user's head makes lateral and/or angularmovements. The inward-facing camera may be any of the CAMs and/or VCAMsdescribed in this disclosure. Optionally, the inward-facing cameraweighs less than 10 g, 5 g or 1 g. Optionally, the inward-facing camerais a thermal camera based on a thermopile sensor, a pyroelectric sensor,or a microbolometer sensor, which may be a FPA sensor.

In one embodiment, the inward-facing camera includes a multi-pixelsensor and a lens, and the sensor plane is tilted by more than 2°relative to the lens plane according to the Scheimpflug principle inorder to capture sharper images when the body is attached to theeyeglasses that are worn by a user.

The clip-one device may include additional components that are fixed toit. In one embodiment, the clip-on device include a wirelesscommunication module fixed to the body which transmits measurements(e.g., images and/or thermal measurements) taken by one or more of thecameras that are fixed to the body. Optionally, the clip-on device mayinclude a battery fixed to the body, which provides power to one or morecomponents fixed to the body. Optionally, the clip-on device may includea processor that controls the operation of one or more of the componentsfixed to the body and/or processes measurements taken by the camerafixed to the body.

In some embodiments, a computer receives measurements taken by theinward-facing camera (and possibly other cameras fixed to the body), andutilizes the measurements to detect a physiological response.Optionally, the computer is not fixed to the body. For example, thecomputer may belong to a device of the user (e.g., a smartphone or asmartwatch), or the computer may be a cloud-based server. Optionally,the computer receives, over a wireless channel, the measurements, whichare sent by the wireless communication module.

The following are various examples of embodiments using different typesof inward- and outward-facing cameras that are fixed to the body, whichmay be used to take images of various regions of interest on the face ofthe user who wears the eyeglasses. It is to be noted that while thediscussion below generally refers to a single “inward-facing camera”and/or a single “outward-facing camera”, embodiments of the clip-ondevice may include multiple inward- and/or outward-facing cameras.

In some embodiments, the inward-facing camera is a thermal camera.Optionally, when the body is attached to the eyeglasses, the thermalcamera is located less than 5 cm from the user's face. Optionally,measurements taken by the thermal camera are transmitted by the wirelesscommunication module and are received by a computer that uses them todetect a physiological response of the user. In one example, when thebody is attached to the eyeglasses, the optical axis of the thermalcamera is above 20° from the Frankfort horizontal plane, and the thermalcamera takes thermal measurements of a region on the user's forehead. Inanother example, when the body is attached to the eyeglasses, thethermal camera takes thermal measurements of a region on the user'snose. In yet another example, when the body is attached to theeyeglasses, the thermal camera takes thermal measurements of a region ona periorbital area of the user.

In one embodiment, the inward-facing camera is a thermal camera. Whenthe body is attached to the eyeglasses, the thermal camera is locatedbelow eye-level of a user who wears the eyeglasses and at least 2 cmfrom the vertical symmetry axis that divides the user's face (i.e., theaxis the goes down the center of the user's forehead and nose).Additionally, when the body is attached to the eyeglasses, theinward-facing thermal camera takes thermal measurements of a region onat least one of the following parts of the user's face: upper lip, lips,and a cheek. Optionally, measurements taken by the thermal camera aretransmitted by the wireless communication module and are received by acomputer that uses them to detect a physiological response of the user.

In another embodiment, the inward-facing camera is a visible-lightcamera. Optionally, when the body is attached to the eyeglasses, thevisible-light camera is located less than 10 cm from the user's face.Optionally, images taken by the visible-light camera are transmitted bythe wireless communication module and are received by a computer thatuses them to detect a physiological response of the user. Optionally,the computer detects the physiological response based on facial skincolor changes (FSCC) that are recognizable in the images. In oneexample, when the body is attached to the eyeglasses, the optical axisof the visible-light camera is above 20° from the Frankfort horizontalplane, and the visible-light camera takes images of a region locatedabove the user's eyes. In another example, when the body is attached tothe eyeglasses, the visible-light camera takes images of a region on thenose of a user who wears the eyeglasses. In still another example, thecomputer detects the physiological response based on facial expressions,and when the body is attached to the eyeglasses, the visible-lightcamera takes images of a region above or below the user's eyes.

In still another embodiment, the inward-facing camera is a visible-lightcamera, and when the body is attached to the eyeglasses, thevisible-light camera takes images of a region on an eye (IM_(E)) of auser who wears the eyeglasses, and is located less than 10 cm from theuser's face. Optionally, the images are transmitted by the wirelesscommunication module and are received by a computer that detects aphysiological response based in IM_(E).

In one example, the computer detects the physiological response based oncolor changes to certain parts of the eye, such as the sclera and/or theiris. Due to the many blood vessels that are close to the surface of theeye, physiological responses that are manifested through changes to theblood flow (e.g., a cardiac pulse and certain emotional responses), maycause recognizable changes to the color of the certain parts of the eye.The various techniques described in this disclosure for detecting aphysiological response based on FSCC that is recognizable in images canbe applied by one skilled in the art to detect a physiological responsebased on color changes to the sclera and/or iris; while the sclera andiris are not the same color as a person's skin, they too exhibit bloodflow-related color changes that are qualitatively similar to FSCC, andthus may be analyzed using similar techniques to the techniques used toanalyze FSCC involving the forehead, nose, and/or cheeks.

In another example, IM_(E) may be utilized to determine the size of thepupil, which may be utilized by the computer to detect certain emotionalresponses (such as based on the assumption that the pupil's responsereflects emotional arousal associated with increased sympatheticactivity).

If needed as part of the computer's detection of the physiologicalresponse, identifying which portions of IM_(E) correspond to certainparts of the eye (e.g., the sclera or iris) can be done utilizingvarious image processing techniques known in the art. For example,identifying the iris and pupil size may be done using the techniquesdescribed in US patent application 20060147094, or in Hayes, Taylor R.,and Alexander A. Petrov. “Mapping and correcting the influence of gazeposition on pupil size measurements.” Behavior Research Methods 48.2(2016): 510-527. Additionally, due to the distinct color differencesbetween the skin, the iris, and the sclera, identification of the irisand/or the white sclera can be easily done by image processing methodsknown in the art.

In one embodiment, the inward-facing camera is a visible-light camera;when the body is attached to the eyeglasses, the visible-light camera islocated below eye-level of a user who wears the eyeglasses, and at least2 cm from the vertical symmetry axis that divides the user's face. Thevisible-light camera takes images (IM_(ROI)) of a region on the upperlip, lips, and/or a cheek. Optionally, IM_(ROI) are transmitted by thewireless communication module and are received by a computer that usesthem to detect a physiological response. In one example, thephysiological response is an emotional response, which is detected basedon extracting facial expressions from IM_(ROI). In another example, thephysiological response is an emotional response, which is detected basedon FSCC recognizable in IM_(ROI). In still another example, thephysiological response, which is detected based FSCC recognizable inIM_(ROI), is heart rate and/or breathing rate.

The body may include an outward-facing camera that may be utilized toprovide measurements that may be used to account for variousenvironmental interferences that can decrease detections of thephysiological response of a user who wears the eyeglasses. Optionally,the outward-facing camera is a head-mounted camera. Optionally, theoutward-facing camera is fixed to the body.

In one embodiment, the inward-facing camera is a thermal camera, andwhen the body is attached to the eyeglasses, the thermal camera islocated less than 10 cm from the face of the user who wears theeyeglasses, and takes thermal measurements of a region of interest(TH_(ROI)) on the face of the user. In this embodiment, anoutward-facing head-mounted thermal camera takes thermal measurements ofthe environment (TH_(ENV)). The wireless communication module transmitsTH_(ROI) and TH_(ENV) to a computer that detects a physiologicalresponse of the user based on TH_(ROI) and TH_(ENV). Optionally, thecomputer utilizes TH_(ENV) to account for thermal interferences from theenvironment, as discussed elsewhere herein.

In another embodiment, the inward-facing camera is a visible-lightcamera, and when the body is attached to the eyeglasses, thevisible-light camera is located less than 10 cm from the face of theuser who wears the eyeglasses and takes images of a region of interest(IM_(ROI)) on the face of the user. In this embodiment, anoutward-facing head-mounted visible-light camera takes images of theenvironment (IM_(ENV)). The wireless communication module transmitsIM_(ROI) and IM_(ENV) to a computer that detects a physiologicalresponse of the user based on IM_(ROI) and IM_(ENV). Optionally, thecomputer detects the physiological response based on FSCC recognizablein IM_(ROI), and utilizes IM_(ENV) to account for variations in ambientlight, as discussed elsewhere herein.

Inward-facing cameras attached to the body may be utilized foradditional purposes, beyond detection of physiological responses. In oneembodiment, the inward-facing camera is a visible-light camera, and theclip-on device includes a second visible-light camera that is also fixedto the body. Optionally, the visible-light camera and/or the secondvisible-light camera are light field cameras. Optionally, when the bodyis attached to the eyeglasses, the first and second visible-lightcameras are located less than 10 cm from the user's face, and takeimages of a first region above eye-level and a second region on theupper lip (IM_(ROI) and IM_(ROI2), respectively). Optionally, thewireless communication module transmits IM_(ROI) and IM_(ROI2) to acomputer that generates an avatar of the user based on IM_(ROI) andIM_(ROI2). Some of the various approaches that may be utilized togenerate the avatar based on IM_(ROI) and IM_(ROI2) are described inco-pending US patent publication 2016/0360970.

Different embodiments of the clip-on device may involve devices ofvarious shapes, sizes, and/or locations of attachment to the eyeglasses.FIG. 14a to FIG. 18 illustrate some examples of clip-on devices. Whenthe body is attached to the eyeglasses, most of the clip-on device maybe located in front of the frame of the eyeglasses, as illustrated inFIG. 14b , FIG. 15b , and FIG. 18, or alternatively, most of the clip-ondevice may be located behind the frame, as illustrated in FIG. 16b andFIG. 17b . Some clip-on devices may include a single unit, such asillustrated in FIG. 15a and FIG. 17a . While other clip-on devices mayinclude multiple units (which each may optionally be considered aseparate clip-on device). Examples of multiple units being attached tothe frame are illustrated in FIG. 14b , FIG. 16b , and FIG. 18. Thefollowing is a more detailed discussion regarding embodimentsillustrated in the figures mentioned above.

FIG. 14a , FIG. 14b , and FIG. 14c illustrate two right and left clip-ondevices comprising bodies 141 and 142, respectively, which areconfigured to attached/detached from an eyeglasses frame 140. The body142 has multiple inward-facing cameras fixed to it, such as camera 143that points at a region on the lower part of the face (such as the upperlip, mouth, nose, and/or cheek), and camera 144 that points at theforehead. The body 142 may include other electronics 145, such as aprocessor, a battery, and/or a wireless communication module. The bodies141 and 142 of the left and right clip-on devices may include additionalcameras illustrated in the drawings as black circles.

In one another embodiment, the eyeglasses include left and right lenses,and when the body is attached to the eyeglasses, most of the volume ofthe clip-on device is located to the left of the left lens or to theright of the right lens. Optionally, the inward-facing camera takesimages of at least one of: a region on the nose of a user wearing theeyeglasses, and a region on the mouth of the user. Optionally, a portionof the clip-on device that is located to the left of the left lens or tothe right of the right lens does not obstruct the sight of the user whenlooking forward.

FIG. 15a and FIG. 15b illustrate a clip-on device that includes a body150, to which two head-mounted cameras are fixed: a head-mounted camera148 that points at a region on the lower part of the face (such as thenose), and a head-mounted camera 149 that points at the forehead. Theother electronics (such as a processor, a battery, and/or a wirelesscommunication module) are located inside the body 150. The clip-ondevice is attached and detached from the frame of the eyeglasses withthe clasp 147.

In one embodiment, when the body is attached to the eyeglasses, most ofthe volume of the clip-on device is located above the lenses of theeyeglasses, and the inward-facing camera takes images of a region on theforehead of a user who wears the eyeglasses. Optionally, a portion ofthe clip-on device that is located above the lenses of the eyeglassesdoes not obstruct the sight of the user when looking forward.

While the clip-on device may often have a design intended to reduce theextent to which it sticks out beyond the frame, in some embodiments, theclip-on device may include various protruding arms. Optionally, thesearms may be utilized in order to position one or more cameras in aposition suitable for taking images of certain regions of the face. FIG.18 illustrates right and left clip-on devices that include bodies 153and 154, respectively, which are configured to attached/detached from aneyeglasses frame. These bodies have protruding arms that hold thehead-mounted cameras. Head-mounted camera 155 measures a region on thelower part of the face, head-mounted camera 156 measures regions on theforehead. The left clip-on device also includes other electronics 157(such as a processor, a battery, and/or a wireless communicationmodule). The clip-on devices illustrated in this figure may includeadditional cameras illustrated in the drawings as black circles.

In other embodiments, at least a certain portion of the clip-on deviceis located behind the eyeglasses' frame. Thus, when the clip-on deviceis attached to the eyeglasses, they may remain aesthetically pleasing,and attaching the clip-on device may cause little or no blocking of theuser's vision. FIG. 16b and FIG. 17b illustrate two examples of clip-ondevices that are mostly attached behind the frame. The following aresome additional examples of embodiments in which a portion of theclip-on device may be located behind the frame.

FIG. 16a and FIG. 16b illustrate two, right and left, clip-on deviceswith bodies 160 and 161, respectively, configured to be attached behindan eyeglasses frame 165. The body 160 has various components fixed to itwhich include: an inward-facing head-mounted camera 162 pointed at aregion below eye-level (such as the upper lip, mouth, nose, and/orcheek), an inward-facing head-mounted camera 163 pointed at a regionabove eye-level (such as the forehead), and other electronics 164 (suchas a processor, a battery, and/or a wireless communication module). Theright and left clip-on devices may include additional camerasillustrated in the drawings as black circles.

FIG. 17a and FIG. 17b illustrate a single-unit clip-on device thatincludes the body 170, which is configured to be attached behind theeyeglasses frame 176. The body 170 has various cameras fixed to it, suchas head-mounted cameras 171 and 172 that are pointed at regions on thelower part of the face (such as the upper lip, mouth, nose, and/orcheek), and head-mounted cameras 173 and 174 that are pointed at theforehead. The spring 175 is configured to apply force that holds thebody 170 to the frame 176. Other electronics 177 (such as a processor, abattery, and/or a wireless communication module), may also be fixed tothe body 170. The clip-on device may include additional camerasillustrated in the drawings as black circles.

In one embodiment, when the body is attached to the eyeglasses, morethan 50% of the out-facing surface of the clip-on device is locatedbehind the eyeglasses frame. Optionally, a portion of the clip-on devicethat is located behind the eyeglasses frame is occluded from a viewerpositioned directly opposite to the eyeglasses, at the same height asthe eyeglasses. Thus, a portion of the clip-on device that is behind theframe might not be visible to other people from many angles, which canmake the clip-on device less conspicuous and/or more aestheticallypleasing. Optionally, a larger portion of the clip-on device is behindthe frame when the body is attached to the eyeglasses, such as more than75% or 90% of the out-facing surface.

Various physiological responses may be detected based on Facial skincolor changes (FSCC) that occur on a user's face. In one embodiment, asystem configured to detect a physiological response based on FSCCincludes at least an inward-facing head-mounted visible-light camera(VCAM_(in)) and a computer. The system may optionally include additionalelements such as a frame and additional inward-facing camera(s) and/oroutward-facing camera(s).

FIG. 24 illustrates one embodiment of the system configured to detect aphysiological response based on FSCC. The system includes a frame 735(e.g., an eyeglasses frame) to which various cameras are physicallycoupled. These cameras include visible-light cameras 740, 741, 742, and743, which may each take images of regions on the user's cheeks and/ornose. Each of these cameras may possibly be VCAM_(in) which is discussedin more detail below. Another possibility for VCAM_(in) is camera 745that takes images of a region on the user's forehead and is coupled tothe upper portion of the frame. Visible-light camera 737, which takesimages of the environment (IM_(ENV)), is an example of VCAM_(out)discussed below, which may optionally be included in some embodiments.Additional cameras that may optionally be included in some embodimentsare outward-facing thermal camera 738 (which may be used to takeTH_(ENV) mentioned below) and inward-facing thermal camera 739 (whichmay be used to take TH_(ROI2) mentioned below).

VCAM_(in) is worn on the user's head and takes images of a region ofinterest (IM_(ROI)) on the user's face. Depending on the physiologicalresponse being detected, the ROI may cover various regions on the user'sface. In one example, the ROI is on a cheek of the user, a region on theuser's nose, and/or a region on the user's forehead. Optionally,VCAM_(in) does not occlude the ROI, is located less than 10 cm from theuser's face, and weighs below 10 g. The ROI is illuminated by ambientlight. Optionally, the system does not occlude the ROI, and the ROI isnot illuminated by a head-mounted light source. Alternatively, the ROImay be illuminated by a head-mounted light source that is weaker thanthe ambient light.

The computer detects the physiological response based on IM_(ROI) byrelying on effects of FSCC that are recognizable in IM_(ROI). Herein,sentences of the form “FSCC recognizable in IM_(ROI)” refer to effectsof FSCC that may be identified and/or utilized by the computer, whichare usually not recognized by the naked eye. The FSCC phenomenon may beutilized to detect various types of physiological responses. In oneembodiment, the physiological response that is detected may involve anexpression of emotional response of the user. For example, the computermay detect whether the user's emotional response is neutral, positive,or negative. In another example, the computer may detect an emotionalresponse that falls into a more specific category such as distress,happiness, anxiousness, sadness, frustration, intrigue, joy, disgust,anger, etc. Optionally, the expression of the emotional response mayinvolve the user making a facial expression and/or a microexpression(whose occurrence may optionally be detected based on IM_(ROI)). Inanother embodiment, detecting the physiological response involvesdetermining one or more physiological signals of the user, such as aheart rate (which may also be referred to as “cardiac pulse”), heartrate variability, and/or a breathing rate.

IM_(ROI) are images generated based on ambient light illumination thatis reflected from the user's face. Variations in the reflected ambientlight may cause FSCC that are unrelated to the physiological responsebeing detected, and thus possibly lead to errors in the detection of thephysiological response. In some embodiments, the system includes anoutward-facing head-mounted visible-light camera (VCAM_(out)), which isworn on the user's head, and takes images of the environment (IM_(ENV)).Optionally, VCAM_(out) is located less than 10 cm from the user's faceand weighs below 10 g. Optionally, VCAM_(out) may include optics thatprovide it with a wide field of view. Optionally, the computer detectsthe physiological response based on both IM_(ROI) and IM_(ENV). Giventhat IM_(ENV) is indicative of illumination towards the face andIM_(ROI) is indicative of reflections from the face, utilizing IM_(ENV)in the detection of the physiological response can account, at least inpart, for variations in ambient light that, when left unaccounted, maypossibly lead to errors in detection of the physiological response.

It is noted that the system may include multiple VCAM_(in) configured totake images of various ROIs on the face, IM_(ROI) may include imagestaken from the multiple VCAM_(in), and multiple VCAM_(out) located atdifferent locations and/or orientation relative to the face may be usedto take images of the environment.

In some embodiments, VCAM_(in) and/or VCAM_(out) are physically coupledto a frame, such as an eyeglasses frame or an augmented realty deviceframe. Optionally, the angle between the optical axes of VCAM_(in) andVCAM_(out) is known to the computer, and may be utilized in thedetection of the physiological response. Optionally, the angle betweenthe optical axes of VCAM_(in) and VCAM_(out) is fixed.

Due to the proximity of VCAM_(in) to the face, in some embodiments,there may be an acute angle between the optical axis of VCAM_(in) andthe ROI (e.g., when the ROI includes a region on the forehead). In orderto improve the sharpness of IM_(ROI), VCAM_(in) may be configured tooperate in a way that takes advantage of the Scheimpflug principle. Inone embodiment, VCAM_(in) includes a sensor and a lens; the sensor planeis tilted by a fixed angle greater than 2° relative to the lens planeaccording to the Scheimpflug principle in order to capture a sharperimage when VCAM_(in) is worn by the user (where the lens plane refers toa plane that is perpendicular to the optical axis of the lens, which mayinclude one or more lenses). Optionally, VCAM_(in) does not occlude theROI. In another embodiment, VCAM_(in) includes a sensor, a lens, and amotor; the motor tilts the lens relative to the sensor according to theScheimpflug principle. The tilt improves the sharpness of IM_(ROI) whenVCAM_(in) is worn by the user.

In addition to capturing images in the visible spectrum, someembodiments may involve capturing light in the near infrared spectrum(NIR). In some embodiments, VCAM_(in) and/or VCAM_(out) may includeoptics and sensors that capture light rays in at least one of thefollowing NIR spectrum intervals: 700-800 nm, 700-900 nm, 700-1,000 nm.Optionally, the computer may utilize data obtained in a NIR spectruminterval to detect the physiological response (in addition to or insteadof data obtained from the visible spectrum). Optionally, the sensors maybe CCD sensors designed to be sensitive in the NIR spectrum and/or CMOSsensors designed to be sensitive in the NIR spectrum.

One advantage of having VCAM_(in) coupled to the frame involves thehandling of chromatic aberrations. Chromatic aberrations refractdifferent wavelengths of light at different angles, depending on theincident angle. When VCAM_(in) is physically coupled to the frame, theangle between VCAM_(in) and the ROI is known, and thus the computer maybe able to select certain subsets of pixels, which are expected tomeasure light of certain wavelengths from the ROI. In one embodiment,VCAM_(in) includes a lens and a sensor comprising pixels; the lensgenerates chromatic aberrations that refract red and blue light rays indifferent angles; the computer selects, based on the angle between thecamera and the ROI (when the user wears the frame), a first subset ofpixels to measure the blue light rays reflected from the ROI, and asecond subset of pixels to measure the red light rays reflected from theROI. Optionally, the first and second subsets are not the same.Optionally, VCAM_(in) may include a sensor that captures light rays alsoin a portion of the NIR spectrum, and the computer selects, based on theangle between VCAM_(in) and the ROI, a third subset of pixels to measurethe NIR light rays reflected from the ROI. Optionally, the second andthird subsets are not the same.

The computer may utilize various approaches in order to detect thephysiological response based on IM_(ROI). Some examples of how such adetection may be implemented are provided in the prior art referencesmentioned above, which rely on FSCC to detect the physiologicalresponse. It is to be noted that while the prior art approaches involveanalysis of video obtained from cameras that are not head-mounted, aretypically more distant from the ROI than VCAM_(in), and are possibly atdifferent orientations relative to the ROI, the computational approachesdescribed in the prior art used to detect physiological responses can bereadily adapted by one skilled in the art to handle IM_(ROI). In somecases, embodiments described herein may provide video in which a desiredsignal is more easily detectable compared to some of the prior artapproaches. For example, given the short distance from VCAM_(in) to theROI, the ROI is expected to cover a larger portion of the images inIM_(ROI) compared to images obtained by video cameras in some of theprior art references. Additionally, due to the proximity of VCAM_(in) tothe ROI, additional illumination that is required in some prior artapproaches, such as illuminating the skin for a pulse oximeter to obtaina photoplethysmographic (PPG) signal, may not be needed. Furthermore,given VCAM_(in)'s fixed location and orientation relative to the ROI(even when the user makes lateral and/or angular movements), manypre-processing steps that need to be implemented by the prior artapproaches, such as image registration and/or face tracking, areextremely simplified in embodiments described herein, or may be foregonealtogether.

IM_(ROI) may undergo various preprocessing steps prior to being used bythe computer to detect the physiological response and/or as part of theprocess of the detection of the physiological response. Somenon-limiting examples of the preprocessing include: normalization ofpixel intensities (e.g., to obtain a zero-mean unit variance time seriessignal), and conditioning a time series signal by constructing a squarewave, a sine wave, or a user defined shape, such as that obtained froman ECG signal or a PPG signal as described in U.S. Pat. No. 8,617,081.Additionally or alternatively, some embodiments may involve generatingfeature values based on a single image or a sequence of images. In someexamples, generation of feature values from one or more images mayinvolve utilization of some of the various approaches described in thisdisclosure for generation of high-level and/or low-level image-basedfeatures.

The following is a discussion of some approaches that may be utilized bythe computer to detect the physiological response based on IM_(ROI).Additionally, implementation-related details may be found in theprovided references and the references cited therein. Optionally,IM_(ENV) may also be utilized by the computer to detect thephysiological response (in addition to IM_(ROI)), as explained in moredetail below.

In some embodiments, the physiological response may be detected usingsignal processing and/or analytical approaches. Optionally, theseapproaches may be used for detecting repetitive physiological signals(e.g., a heart rate, heart rate variability, or a breathing rate) inIM_(ROI) taken during a certain period. Optionally, the detectedphysiological response represents the value of the physiological signalof the user during the certain period.

In one example, U.S. Pat. No. 8,768,438, titled “Determining cardiacarrhythmia from a video of a subject being monitored for cardiacfunction”, describes how a heart rate may be determined based on FSCC,which are represented in a PPG signal obtained from video of the user.In this example, a time series signal is generated from video images ofa subject's exposed skin, and a reference signal is used to perform aconstrained source separation (which is a variant of ICA) on the timeseries signals to obtain the PPG signal. Peak-to-peak pulse points aredetected in the PPG signal, which may be analyzed to determineparameters such as heart rate, heart rate variability, and/or to obtainpeak-to-peak pulse dynamics that can be indicative of conditions such ascardiac arrhythmia.

In another example, U.S. Pat. No. 8,977,347, titled “Video-basedestimation of heart rate variability”, describes how a times-seriessignal similar to the one described above may be subjected to adifferent type of analysis to detect the heart rate variability. In thisexample, the time series data are de-trended to remove slownon-stationary trends from the signal and filtered (e.g., using bandpassfiltering). Following that, low frequency and high frequency componentsof the integrated power spectrum within the time series signal areextracted using Fast Fourier Transform (FFT). A ratio of the low andhigh frequency of the integrated power spectrum within these componentsis computed. And analysis of the dynamics of this ratio over time isused to estimate heart rate variability.

In yet another example, U.S. Pat. No. 9,020,185, titled “Systems andmethods for non-contact heart rate sensing”, describes how atimes-series signals obtained from video of a user can be filtered andprocessed to separate an underlying pulsing signal by, for example,using an ICA algorithm. The separated pulsing signal from the algorithmcan be transformed into frequency spacing data using FFT, in which theheart rate can be extracted or estimated.

In some embodiments, the physiological response may be detected usingmachine learning-based methods. Optionally, these approaches may be usedfor detecting expressions of emotions and/or values of physiologicalsignals.

Generally, machine learning-based approaches involve training a model onsamples, with each sample including: feature values generated based onIM_(ROI) taken during a certain period, and a label indicative of thephysiological response during the certain period. Optionally, the modelmay be personalized for a user by training the model on samplesincluding: feature values generated based on IM_(ROI) of the user, andcorresponding labels indicative of the user's respective physiologicalresponses. Some of the feature values in a sample may be generated basedon other sources of data (besides IM_(ROI)), such as measurements of theuser generated using thermal cameras, movement sensors, and/or otherphysiological sensors, and/or measurements of the environment.Optionally, IM_(ROI) of the user taken during an earlier period mayserve as a baseline to which to compare. Optionally, some of the featurevalues may include indications of confounding factors, which may affectFSCC, but are unrelated to the physiological response being detected.Some examples of confounding factors include touching the face, thermalradiation directed at the face, and consuming certain substances such asa medication, alcohol, caffeine, or nicotine.

Training the model may involve utilization of various trainingalgorithms known in the art (e.g., algorithms for training neuralnetworks and/or other approaches described herein). After the model istrained, feature values may be generated for IM_(ROI) for which thelabel (physiological response) is unknown, and the computer can utilizethe model to detect the physiological response based on these featurevalues.

It is to be noted that in some embodiments, the model is trained basedon data that includes measurements of the user, in which case it may beconsidered a personalized model of the user. In other embodiments, themodel is trained based on data that includes measurements of one or moreother users, in which case it may be considered a general model.

In order to achieve a robust model, which may be useful for detectingthe physiological response in various conditions, in some embodiments,the samples used in the training may include samples based on IM_(ROI)taken in different conditions and include samples with various labels(e.g., expressing or not expressing certain emotions, or differentvalues of physiological signals). Optionally, the samples are generatedbased on IM_(ROI) taken on different days.

The following are four examples of different compositions of samplesthat may be used when training the model in different embodiments. The“measured user” in the four examples below may be “the user” who ismentioned above (e.g., when the model is a personalized model that wastrained on data that includes measurements of the user), or a user fromamong one or more other users (e.g., when the model is a general modelthat was trained on data that includes measurements of the other users).In a first example, the system does not occlude the ROI, and the modelis trained on samples generated from a first set of IM_(ROI) taken whilethe measured user was indoors and not in direct sunlight, and is alsotrained on other samples generated from a second set of IM_(ROI) takenwhile the measured user was outdoors, in direct sunlight. In a secondexample, the model is trained on samples generated from a first set ofIM_(ROI) taken during daytime, and is also trained on other samplesgenerated from a second set of IM_(ROI) taken during nighttime. In athird example, the model is trained on samples generated from a firstset of IM_(ROI) taken while the measured user was exercising and moving,and is also trained on other samples generated from a second set ofIM_(ROI) taken while the measured user was sitting and not exercising.And a fourth example, the model is trained on samples generated from afirst set of IM_(ROI) taken less than 30 minutes after the measured userhad an alcoholic beverage, and is also trained on other samplesgenerated from a second set of IM_(ROI) taken on a day in which themeasured user did not have an alcoholic beverage.

Labels for the samples may be obtained from various sources. In oneembodiment, the labels may be obtained utilizing one or more sensorsthat are not VCAM_(in). In one example, a heart rate and/or heart ratevariability may be measured using an ECG sensor. In another example, thebreathing rate may be determined using a smart shirt with sensorsattached to the chest (e.g., a smart shirt by Hexoskin®). In yet anotherexample, a type emotional response of the user may be determined basedon analysis of a facial expression made by the user, analysis of theuser's voice, analysis of thermal measurements of regions of the face ofthe user, and/or analysis of one or more of the followingsensor-measured physiological signals of the user: a heart rate, heartrate variability, breathing rate, and galvanic skin response.

In another embodiment, a label describing an emotional response of theuser may be inferred. In one example, the label may be based on semanticanalysis of a communication of the user, which is indicative of theuser's emotional state at the time IM_(ROI) were taken. In anotherexample, the label may be generated in a process in which the user isexposed to certain content, and a label is determined based on anexpected emotional response corresponding to the certain content (e.g.,happiness is an expected response to a nice image while distress is anexpected response to a disturbing image).

Due to the nature of the physiological responses being detected and thetype of data (video images), a machine learning approach that may beapplied in some embodiments is “deep learning”. In one embodiment, themodel may include parameters describing multiple hidden layers of aneural network. Optionally, the model may include a convolution neuralnetwork (CNN). In one example, the CNN may be utilized to identifycertain patterns in the video images, such as the patterns of thereflected FSCC due to the physiological response. Optionally, detectingthe physiological response may be done based on multiple, possiblysuccessive, images that display a certain pattern of change over time(i.e., across multiple frames), which characterizes the physiologicalresponse being detected. Thus, detecting the physiological response mayinvolve retaining state information that is based on previous images.Optionally, the model may include parameters that describe anarchitecture that supports such a capability. In one example, the modelmay include parameters of a recurrent neural network (RNN), which is aconnectionist model that captures the dynamics of sequences of samplesvia cycles in the network's nodes. This enables RNNs to retain a statethat can represent information from an arbitrarily long context window.In one example, the RNN may be implemented using a long short-termmemory (LSTM) architecture. In another example, the RNN may beimplemented using a bidirectional recurrent neural network architecture(BRNN).

Some of the prior art references mentioned herein provide additionaldetailed examples of machine learning-based approaches that may beutilized to detect the physiological response (especially in the case inwhich it corresponds to an emotional response). In one example, Ramirez,et al. (“Color analysis of facial skin: Detection of emotional state”)describe detection of an emotional state using various machine learningalgorithms including decision trees, multinomial logistic regression,and latent-dynamic conditional random fields. In another example, Wang,et al. (“Micro-expression recognition using color spaces”) describevarious feature extraction methods and pixel color valuetransformations, which are used to generate inputs for a support vectormachine (SVM) classifier trained to identify microexpressions.

As mentioned above, in some embodiments, IM_(ENV) may be utilized in thedetection of the physiological response to account, at least in part,for illumination interferences that may lead to errors in the detectionof the physiological response. There are different ways in whichIM_(ENV) may be utilized for this purpose.

In one embodiment, when variations in IM_(ENV) reach a certain threshold(e.g., which may correspond to ambient light variations above a certainextent), the computer may refrain from detecting the physiologicalresponse.

In another embodiment, IM_(ENV) may be utilized to normalize IM_(ROI)with respect to the ambient light. For example, the intensity of pixelsin IM_(ROI) may be adjusted based on the intensity of pixels in IM_(ENV)when IM_(ROI) were taken. US patent application number 20130215244describes a method of normalization in which values of pixels from aregion that does not contain a signal (e.g., background regions thatinclude a different body part of the user or an object behind the user)are subtracted from regions of the image that contain the signal of thephysiological response. While the computational approach describedtherein may be applied to embodiments in this disclosure, the exactsetup described therein may not work well in some cases due to the closeproximity of VCAM_(in) to the face and the fact that VCAM_(in) ishead-mounted. Thus, it may be advantageous to subtract a signal from theenvironment (IM_(ENV)) that is obtained from VCAM_(out), which may moreaccurately represent the ambient light illuminating the face.

It is to be noted that training data that includes a ground-truth signal(i.e., values of the true physiological response corresponding toIM_(ROI) and IM_(ENV)) may be utilized to optimize the normalizationprocedure used to correct IM_(ROI) with respect to the ambient lightmeasured in IM_(ENV). For example, such optimization may be used todetermine parameter values of a function that performs the subtractionabove, which lead to the most accurate detections of the physiologicalresponse.

In still another embodiment, IM_(ENV) may be utilized to generatefeature values in addition to IM_(ROI). Optionally, at least some of thesame types of feature values generated based on IM_(ROI) may also begenerated based on IM_(ENV). Optionally, at least some of the featurevalues generated based on IM_(ENV) may relate to portions of images,such as average intensity of patches of pixels in IM_(ENV).

By utilizing IM_(ENV) as inputs used for the detection of thephysiological response, a machine learning-based model may be trained tobe robust, and less susceptible, to environmental interferences such asambient light variations. For example, if the training data used totrain the model includes samples in which no physiological response waspresent (e.g., no measured emotional response or microexpression wasmade), but some ambient light variations might have introduced someFSCC-related signal, the model will be trained such that feature valuesbased on IM_(ENV) are used to account for such cases. This can enablethe computer to negate, at least in part, the effects of suchenvironmental interferences, and possibly make more accurate detectionsof the physiological response.

In one embodiment, the computer receives an indication indicative of theuser consuming a confounding substance that is expected to affect FSCC(e.g., alcohol, drugs, certain medications, and/or cigarettes). Thecomputer detects the physiological response, while the consumedconfounding substance affects FSCC, based on: IM_(ROI), the indication,and a model that was trained on: a first set of IM_(ROI) taken while theconfounding substance affected FSCC, and a second set of IM_(ROI) takenwhile the confounding substance did not affect FSCC.

Prior art FSCC systems are sensitive to user movements and do notoperate well while the user is running. This is because state-of-the-artFSCC systems use hardware and automatic image trackers that are notaccurate enough to crop correctly the ROI from the entire image whilerunning, and the large errors in cropping the ROI are detrimental to theperformances of the FSCC algorithms. Contrary to the prior art FSCCsystems, the disclosed VCAM_(in) remains pointed at its ROI also whenthe user's head makes angular and lateral movements, and thus thecomplicated challenges related to image registration and ROI trackingare much simplified or even eliminated. Therefore, systems based onVCAM_(in) (such as the one illustrated in FIG. 24) may detect thephysiological response (based on FSCC) also while the user is running.

VCAM_(in) may be pointed at different regions on the face. In a firstembodiment, the ROI is on the forehead, VCAM_(in) is located less than10 cm from the user's face, and optionally the optical axis of VCAM_(in)is above 20° from the Frankfort horizontal plane. In a secondembodiment, the ROI is on the nose, and VCAM_(in) is located less than10 cm from the user's face. Because VCAM_(in) is located close to theface, it is possible to calculate the FSCC based on a small ROI, whichis irrelevant to the non-head-mounted prior arts that are limited by theaccuracy of their automatic image tracker. In a third embodiment,VCAM_(in) is pointed at an eye of the user. The computer selects thesclera as the ROI and detects the physiological response based on colorchanges recognizable in IM_(ROI) of the sclera. In a fourth embodiment,VCAM_(in) is pointed at an eye of the user. The computer selects theiris as the ROI and detects the physiological response based on colorchanges recognizable in IM_(ROI) of the iris. Optionally, the computerfurther calculates changes to the pupil diameter based on the IM_(ROI)of the iris, and detects an emotional response of the user based on thechanges to the pupil diameter.

In order to improve the detection accuracy, and in some cases in orderto better account for interferences, the computer may utilizemeasurements of one or more head-mounted thermal cameras in thedetection of the physiological response. In one embodiment, the systemmay include an inward-facing head-mounted thermal camera that takesthermal measurements of a second ROI (TH_(ROI2)) on the user's face.Optionally, ROI and ROI₂ overlap, and the computer utilizes TH_(ROI1) todetect the physiological response. Optionally, on average, detecting thephysiological response based on both FSCC recognizable in IM_(ROI) andTH_(ROI1) is more accurate than detecting the physiological responsebased on the FSCC without TH_(ROI2). Optionally, the computer utilizesTH_(ROI1) to account, at least in part, for temperature changes, whichmay occur due to physical activity and/or consumption of certainmedications that affect the blood flow. Optionally, the computerutilizes TH_(ROI1) by generating feature values based on TH_(ROI2), andutilizing a model that was trained on data comprising TH_(ROI1) in orderto detect the physiological response.

In another embodiment, the system may include an outward-facinghead-mounted thermal camera that takes thermal measurements of theenvironment (TH_(ENV)). Optionally, the computer may utilize TH_(ENV) todetect the physiological response (e.g., by generating feature valuesbased on TH_(ENV) and utilizing a model trained on data comprisingTH_(ENV)). Optionally, on average, detecting the physiological responsebased on both FSCC recognizable in IM_(ROI) and TH_(ENV) is moreaccurate than detecting the physiological response based on the FSCCwithout TH_(ENV). Optionally, the computer utilizes TH_(ENV) to account,at least in part, for thermal interferences from the environment, suchas direct sunlight and/or a nearby heater.

In addition to detecting a physiological response, in some embodiments,the computer may utilize IM_(ROI) to generate an avatar of the user(e.g., in order to represent the user in a virtual environment).Optionally, the avatar may express emotional responses of the user,which are detected based on IM_(ROI). Optionally, the computer maymodify the avatar of the user to show synthesized facial expressionsthat are not manifested in the user's actual facial expressions. In oneembodiment, the synthesized facial expressions correspond to emotionalresponses detected based on FSCC that are recognizable in IM_(ROI). Inanother embodiment, the synthesized facial expressions correspond toemotional responses detected based on thermal measurements taken by CAM.Some of the various approaches that may be utilized to generate theavatar based on IM_(ROI) are described in co-pending US patentpublication 2016/0360970.

The following method for detecting a physiological response based onfacial skin color changes (FSCC) may be used by systems modeledaccording to FIG. 24. The steps described below may be performed byrunning a computer program having instructions for implementing themethod. Optionally, the instructions may be stored on acomputer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps:

In Step 1, taking images of a region of interest (IM_(ROI)) on a user'sface utilizing an inward-facing head-mounted visible-light camera(VCAM_(in)). The ROI is illuminated by ambient light.

And in Step 2, detecting the physiological response based on FSCCrecognizable in IM_(ROI). Optionally, detecting the physiologicalresponse involves generating feature values based on IM_(ROI) andutilizing a model to calculate, based on the feature values, a valueindicative of an extent of the physiological response. Optionally, themodel was trained based on IM_(ROI) of the user taken during differentdays.

In one embodiment, the method may optionally include a step of takingimages of the environment (IM_(ENV)) utilizing an outward-facinghead-mounted visible-light camera (VCAM_(out)). Optionally, detectingthe physiological response is also based on IM_(ENV).

Normally, the lens plane and the sensor plane of a camera are parallel,and the plane of focus (PoF) is parallel to the lens and sensor planes.If a planar object is also parallel to the sensor plane, it can coincidewith the PoF, and the entire object can be captured sharply. If the lensplane is tilted (not parallel) relative to the sensor plane, it will bein focus along a line where it intersects the PoF. The Scheimpflugprinciple is a known geometric rule that describes the orientation ofthe plane of focus of a camera when the lens plane is tilted relative tothe sensor plane.

FIG. 20a is a schematic illustration of an inward-facing head-mountedcamera 550 embedded in an eyeglasses frame 551, which utilizes theScheimpflug principle to improve the sharpness of the image taken by thecamera 550. The camera 550 includes a sensor 558 and a lens 555. Thetilt of the lens 555 relative to sensor 558, which may also beconsidered as the angle between the lens plane 555 and the sensor plane559, is determined according to the expected position of the camera 550relative to the ROI 552 when the user wears the eyeglasses. For arefractive optical lens, the “lens plane” 556 refers to a plane that isperpendicular to the optical axis of the lens 555. Herein, the singularalso includes the plural, and the term “lens” refers to one or morelenses. When “lens” refers to multiple lenses (which is usually the casein most modern cameras having a lens module with multiple lenses), thenthe “lens plane” refers to a plane that is perpendicular to the opticalaxis of the lens module.

The Scheimpflug principle may be used for both thermal cameras (based onlenses and sensors for wavelengths longer than 2500 nm) andvisible-light and/or near-IR cameras (based on lenses and sensors forwavelengths between 400-900 nm). FIG. 20b is a schematic illustration ofa camera that is able to change the relative tilt between its lens andsensor planes according to the Scheimpflug principle. Housing 311 mountsa sensor 312 and lens 313. The lens 313 is tilted relative to the sensor312. The tilt may be fixed according to the expected position of thecamera relative to the ROI when the user wears the HMS, or may beadjusted using motor 314. The motor 314 may move the lens 313 and/or thesensor 312.

In one embodiment, an HMS device includes a frame configured to be wornon a user's head, and an inward-facing camera physically coupled to theframe. The inward-facing camera may assume one of two configurations:(i) the inward-facing camera is oriented such that the optical axis ofthe camera is above the Frankfort horizontal plane and pointed upward tocapture an image of a region of interest (ROI) above the user's eyes, or(ii) the inward-facing camera is oriented such that the optical axis isbelow the Frankfort horizontal plane and pointed downward to capture animage of an ROI below the user's eyes. The inward-facing camera includesa sensor and a lens. The sensor plane is tilted by more than 2° relativeto the lens plane according to the Scheimpflug principle in order tocapture a sharper image.

In another embodiment, an HMS includes an inward-facing head-mountedcamera that captures an image of an ROI on a user's face, when worn onthe user's head. The ROI is on the user's forehead, nose, upper lip,cheek, and/or lips. The camera includes a sensor and a lens. And thesensor plane is tilted by more than 2° relative to the lens planeaccording to the Scheimpflug principle in order to capture a sharperimage.

Because the face is not planar and the inward-facing head-mounted camerais located close to the face, an image captured by a camera having awide field of view (FOV) and a low f-number may not be perfectly sharp,even after applying the Scheimpflug principle. Therefore, in someembodiments, the tilt between the lens plane and the sensor plane isselected such as to adjust the sharpness of the various areas covered inthe ROI according to their importance for detecting the user'sphysiological response (which may be the user's emotional response insome cases). In one embodiment, the ROI covers first and second areas,where the first area includes finer details and/or is more important fordetecting the physiological response than the second area. Therefore,the tilt between the lens and sensor planes is adjusted such that theimage of the first area is shaper than the image of the second area.

In another embodiment, the ROI covers both a first area on the upper lipand a second area on a cheek, and the tilt is adjusted such that theimage of the first area is shaper than the image of the second area,possibly because the upper lip usually provides more information and hasmore details relative to the cheek.

In still another embodiment, the ROI covers both a first area on theupper lip and a second area on the nose, and the tilt is adjusted suchthat the image of the first area is shaper than the image of the secondarea, possibly because the upper lip usually provides more informationrelative to the nose.

In still another embodiment, the ROI covers a first area on the cheekstraight above the upper lip, a second area on the cheek from the edgeof the upper lip towards the ear, and a third area on the nose. And thetilt between the lens plane and the sensor plane is adjusted such thatthe image of the first area is shaper than both the images of the secondand third areas.

In still another embodiment, the ROI covers both a first area on thelips and a second area on the chin, and the tilt is adjusted such thatthe image of the first area is shaper than the image of the second area,possibly because the lips usually provides more information than thechin.

In still another embodiment, the camera is a visible-light camera, andthe ROI covers both a first area on the lower forehead (including aneyebrow) and a second area on the upper forehead, and the tilt isadjusted such that the image of the first area is shaper than the imageof the second area, possibly because the eyebrow provides moreinformation about the user's emotional response than the upper forehead.

In still another embodiment, the camera is a thermal camera, and the ROIcovers an area on the forehead, and the tilt is adjusted such that theimage of a portion of the middle and upper part of the forehead (belowthe hair line) is shaper than the image of a portion of the lower partof the forehead, possibly because the middle and upper parts of theforehead are more indicative of prefrontal cortex activity than thelower part of the forehead, and movements of the eyebrows disturb thethermal measurements of the lower part of the forehead.

In one embodiment, the tilt between the lens plane and sensor plane isfixed. The fixed tilt is selected according to an expected orientationbetween the camera and the ROI when a user wears the frame. Having afixed tilt between the lens and sensor planes may eliminate the need foran adjustable electromechanical tilting mechanism. As a result, a fixedtilt may reduce the weight and cost of the camera, while still providinga sharper image than an image that would be obtained from a similarcamera in which the lens and sensor planes are parallel. The magnitudeof the fixed tilt may be selected according to facial dimensions of anaverage user expected to wear the system, or according to a model of thespecific user expected to wear the system in order to obtain thesharpest image.

In another embodiment, the system includes an adjustableelectromechanical tilting mechanism configured to change the tiltbetween the lens and sensor planes according to the Scheimpflugprinciple based on the orientation between the camera and the ROI whenthe frame is worn by the user. The tilt may be achieved using at leastone motor, such as a brushless DC motor, a stepper motor (without afeedback sensor), a brushed DC electric motor, a piezoelectric motor,and/or a micro-motion motor.

The adjustable electromechanical tilting mechanism configured to changethe tilt between the lens and sensor planes may include one or more ofthe following mechanisms: (i) a mirror that changes its angle; (ii) adevice that changes the angle of the lens relative to the sensor; and/or(iii) a device that changes the angle of the sensor relative to thelens. In one embodiment, the camera, including the adjustableelectromechanical tilting mechanism, weighs less than 10 g, and theadjustable electromechanical tilting mechanism is able to change thetilt in a limited range below 30° between the two utmost orientationsbetween the lens and sensor planes. Optionally, the adjustableelectromechanical tilting mechanism is able to change the tilt in alimited range below 20° between the two utmost orientations between thelens and sensor planes. In another embodiment, the adjustableelectromechanical tilting mechanism is able to change the tilt in alimited range below 10°. In some embodiments, being able to change thetilt in a limited range reduces at least one of the weight, cost, andsize of the camera, which is advantageous for a wearable device. In oneexample, the camera is manufactured with a fixed predetermined tiltbetween the lens and sensor planes, which is in addition to the tiltprovided by the adjustable electromechanical tilting mechanism. Thefixed predetermined orientation may be determined according to theexpected orientation between the camera and the ROI for an average user,such that the adjustable electromechanical tilting mechanism is used tofine-tune the tilt between the lens and sensor planes for the specificuser who wears the frame and has facial dimensions that are differentfrom the average user.

Various types of cameras may be utilized in different embodimentsdescribed herein. In one embodiment, the camera is a thermal camera thattakes thermal measurements of the ROI with a focal plane array thermalsensor having an angle above 2° between the lens and sensor planes.Optionally, the thermal camera weighs below 10 g, is located less than10 cm from the user's face, and the tilt of the lens plane relative tothe sensor plane is fixed. The fixed tilt is selected according to anexpected orientation between the camera and the ROI when the user wearsthe frame. Optionally, the system includes a computer to detect aphysiological response based on the thermal measurements. Optionally,the computer processes time series measurements of each sensing elementindividually to detect the physiological response.

In another embodiment, the camera is a visible-light camera that takesvisible-light images of the ROI, and a computer generates an avatar forthe user based on the visible-light images. Some of the variousapproaches that may be utilized to generate the avatar based on thevisible-light images are described in co-pending US patent publication2016/0360970. Additionally or alternatively, the computer may detect anemotional response of the user based on (i) facial expressions in thevisible-light images utilizing image processing, and/or (ii) facial skincolor changes (FSCC), which result from concentration changes ofhemoglobin and/or oxygenation.

It is to be noted that there are various approaches known in the art foridentifying facial expressions from images. While many of theseapproaches were originally designed for full-face frontal images, thoseskilled in the art will recognize that algorithms designed for full-facefrontal images may be easily adapted to be used with images obtainedusing the inward-facing head-mounted visible-light cameras disclosedherein. For example, the various machine learning techniques describedin prior art references may be applied to feature values extracted fromimages that include portions of the face from orientations that are notdirectly in front of the user. Furthermore, due to the closeness of thevisible-light cameras to the face, facial features are typically largerin images obtained by the systems described herein. Moreover, challengessuch as image registration and face tracking are vastly simplified andpossibly non-existent when using inward-facing head-mounted cameras. Thereference Zeng, Zhihong, et al. “A survey of affect recognition methods:Audio, visual, and spontaneous expressions.” IEEE transactions onpattern analysis and machine intelligence 31.1 (2009): 39-58, describessome of the algorithmic approaches that may be used for this task. Thefollowing references discuss detection of emotional responses based onFSCC: (i) Ramirez, Geovany A., et al. “Color analysis of facial skin:Detection of emotional state” in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops, 2014; and (ii) Wang,Su-Jing, et al. “Micro-expression recognition using color spaces”, inIEEE Transactions on Image Processing 24.12 (2015): 6034-6047.

In still another embodiment, the camera is a light field camera thatimplements a predetermined blurring at a certain Scheimpflug angle, anddecodes the predetermined blurring as function of the certainScheimpflug angle. The light field camera may include an autofocusing ofthe image obtained using the tilting mechanism based on the principlethat scene points that are not in focus are blurred while scene pointsin focus are sharp. The autofocusing may study a small region around agiven pixel; the region is expected to get sharper as the Scheimpflugadjustment gets better, and vice versa. Additionally or alternatively,the autofocusing may use the variance of the neighborhood around eachpixel as a measure of sharpness, where a proper Scheimpflug adjustmentshould increase the variance.

Thermal and/or FSCC patterns corresponding to physiological responsesmay show high variability between different users due to variability ofthe their brains, blood vessel locations, skin properties, hair,physical conditions, and face shapes and sizes. Thus, patterns and/orvarious extractable features from one user's thermal and/or FSCC datamay not be easily transferable to another user, or even to the same userunder different physiological and/or mental conditions. Therefore, someof the embodiments described herein involve training personalized modelsinvolving thermal and/or FSCC patterns that are predictive of varioususer-defined categories of experiencing and/or perceiving certainevents. Personalized models can overcome some of the possibledisadvantages of using normed physiological statistics, which paves theway for personalized training, detection, and therapies, which are ableto account for arbitrary user-defined physiological and/or mental statescorresponding to a wide variety of individual needs. Leveraging machinelearning algorithms can enable assignment of arbitrary user-definedphysiological and/or mental states to recorded thermal and/or FSCC dataduring day-to-day activities, which are later used as basis forautomatic detection and/or therapies for the user, optionally withoutinvolving a clinician.

The personalized model does not need to correspond to a standarduniversally applicable pattern, and thus the user may be free to definehis/her arbitrary user-defined physiological and/or mental states. Inother words, in addition to (or instead of) detecting a state thatcorresponds to some arbitrary population average, the personalized modelallows a personalized detection of a user-defined state.

One embodiment in which a personalized model is utilized involves atraining phase and an operation phase. In the training phase, the systemidentifies desired and/or undesired physiological and/or mental statesof the user using active methods (e.g., the user presses a button)and/or passive methods (e.g., applying semantic analysis to the user'sspeech and typing). The system may also continue to update thepersonalized model to accommodate for changes over time, to supportsincreased efficacy, and to identify new personalized states beyond thoserepresented by population average. Instead of relying on a model trainedbased on data obtained from a wide population, the personalized modelmay decouple commonly coupled ROIs and/or putative physiologicalresponses from the applications, allowing the user to train the systemto detect arbitrary personalized thermal and/or FSCC patterns that maynot suite the wide population. Training the personalized model may bebased on known machine learning methods such as neural networks,supervised machine learning, pattern recognition, pattern matching, etc.The system may detect, predict, and train for the arbitrary user-definedphysiological and/or mental states, identified by personalized thermaland/or FSCC patterns, not limited to averages obtained from a widepopulation.

In the operation phase, the system alerts, predicts, and/or treats theuser based on the personalized model. The system may alert when the useris in the desired/undesired state, predict when the user is going to bein that state, and treat the user to get into a desired state or avoidan undesired state by providing a feedback. The operation phase mayinclude known biofeedback/neurofeedback interactive sessions tuned toguide the user towards the user-defined personalized physiologicaland/or mental states. For example, the personalized model may be trainedto guide the user towards flow, creativity, curiosity, compassion,and/or happiness states, as defined and experienced by the user, and toalert against anger, aggression, boredom, and/or sadness, also asdefined and experienced by the user, without these necessarily beingsuitable for the wide population.

One embodiment of a method for personalized thermal and/or FSCCdetection includes a timestamping step, a machine learning step, arefinement step, an optional detection step, and an optional biofeedbackstep (where biofeedback refers also to neurofeedback).

In the timestamping step, an HMS records arbitrary user-definedphysiological and/or mental states for personal use. The user mayprovide, via a user interface, timestamped markers on the recorded dataused as labels by machine learning approaches for detecting targetuser-defined physiological and/or mental states (which may be desired orundesired states). When the user engages in a certain task, and as theuser enters a target state, the user may (via a user interface) manualprovide a timestamp to mark the time of entering into the target state,and/or the computer may set an automated timestamp based on inferringthe entering into the target state from the user's performance and/oractivities (for example, using predetermined limits of performance thatonce reached automatically trigger timestamping the recorded data asentering into the target state). Upon leaving the target state, the usermay provide a timestamp to mark the leaving of the target state, and/orthe computer may set an automated time stamp based on inferring theleaving of the target state from the user's performance and/oractivities. Several iterations involving timestamping of entering andleaving the target state may complete the timestamping step.

In the machine learning step, the computer extracts and selects featuresfrom the thermal and/or FSCC measurements, labels the extracted andselected features according to the timestamps, and tries one or moremachine learning algorithms to train a classifier, while treating themeasurements as the training and testing sets. Optionally, for uniquepersonalized states, the machine learning algorithm may be optimized forcross-validation by splitting the training set into a first part usedfor training and a second part used for testing. In addition, testingsets comprising data of other users may be used to measure theclassifier's generality. The following examples illustrate various waysto label the HMS measurements based on the timestamps.

In a first example, the computer may (i) label as “not desired” TH_(ROI)taken before receiving from the user a first timestamp marking theentering into a desired state, (ii) label as “desired” TH_(ROI) takenafter receiving the first timestamp and before receiving a secondtimestamp marking the leaving of the desired state, and (iii) label as“not desired” TH_(ROI) taken after receiving the second timestamp.Optionally, the computer may label as “unknown” TH_(ROI) takensufficiently before receiving the first timestamp and TH_(ROI) takensufficiently after receiving the second timestamp.

In a second example, the computer may (i) label as “leading to headache”TH_(ROI) taken during a first window of time before receiving from theuser a first timestamp marking occurrence of a headache, (ii) label as“headache” TH_(ROI) taken after receiving the first timestamp and untila second window before receiving from the user a second timestampmarking “no headache”, (iii) label as “headache leaving” TH_(ROI) takenduring the second window, and (iv) label as “no headache” TH_(ROI) takenafter receiving the second timestamp.

In a third example, the computer may (i) label as “leading to asthmaattack” TH_(ROI) breath indicative of the user's breathing pattern (suchas thermal measurements of a region on the upper lip) taken during afirst window before identifying that the user uses a first inhaler, (ii)label as “first inhaler immediate effect” TH_(breath) taken during asecond window after using the first inhaler, (iii) label as “firstinhaler long effect” TH_(breath) taken during a third window followingthe second window, and (iv) label as “second inhaler immediate effect”TH_(breath) taken during a fourth window after identifying that the useruses a second inhaler. Optionally, the computer may use the automatedlabeling for assessing the user's reaction to using the first inhaler vsusing the second inhaler.

In a fourth example, the computer may (i) label as “buildingconcentration” TH_(breath) indicative of the user's breathing patternand TH_(forehead) indicative of a thermal pattern on the user's foreheadtaken while the user's software agent indicates that the user does notcheck distracting websites (such as social networks, news and email) butthe user's gaze is not essentially continuously focused on the screen,(ii) label as “concentrated” TH_(breath) and TH_(forehead) taken whilethe software agent indicates that the user's gaze is continuouslyfocused on the screen and until a certain duration before the user lostconcentration, and (iii) label as “start losing concentration”TH_(breath) and TH_(forehead) taken during the certain duration.

In a fifth example, the computer may (i) label as “possibly happy”TH_(ROI) and FSCC taken during a first window before a speech analysismodule provides a timestamp that the user is happy, (ii) label as“happy” TH_(ROI) and FSCC taken during a second window after receivingthe timestamp, and (iii) label as “angry” TH_(ROI) and FSCC taken duringa third window after the speech analysis module provides a timestampthat the user is angry.

In the refinement step, the computer starts guessing the physiologicaland/or mental states, and asks the user to confirm correct, incorrect,or inapplicable status of the guesses. The refinement step increasesfidelity the more it is performed.

In the optional detection step, the computer analyzes in real timefeature values generated based on the thermal and/or FSCC measurementsin order to alert the user about entering and/or leaving a target state.For example, the computer permits administration of pain medication tothe user after the classifier determines that the user experiences painabove a threshold previously determined by the user during thetimestamping step. This may reduce addiction by reducing unnecessaryadministrations of higher dose pain medication. Additionally, the usermay be trained to control his/her pain perception during the biofeedbackstep, which may be more effective after a personalized model has beenapplied.

In the optional biofeedback step, the computer generates a feedback forthe user based on the personalized target state. The biofeedback stepmay use a standard biofeedback protocol, but instead of training theuser towards achieving externally derived thermal and/or FSCC targetpatterns that suit the wide population, the user is trained to achievepersonalized thermal and/or FSCC target patterns that most closelyresemble the thermal and/or FSCC patterns found to be predictive duringthe timestamping and refinement steps.

In one embodiment, the user labels during the timestamping step pairs ofundesired and desired states (such as pain vs no pain, migraine vs nomigraine, angry vs calmed, stressed vs calmed, concentrated vs notconcentrated, sad vs happy, self-focused vs compassionate). Then thebiofeedback step trains the user to move out of the undesired state by(i) encouraging changes that bring the current measured thermal and/orFSCC pattern closer to the desired personalized thermal and/or FSCCpattern found to be predictive during the timestamping and refinementsteps, and (ii) discouraging changes that bring the current measuredthermal and/or FSCC pattern closer to the undesired personalized thermaland/or FSCC pattern found to be predictive during the timestamping andrefinement steps.

The following is one example of the information flow in an HMS thatincludes a head-mounted thermal camera and a computer. In thetimestamping step, the head-mounted thermal camera takes thermalmeasurements, and the user (or computer) adds manual (or automated)timestamps for entering and/or leaving a target state. The timestampingstep feeds the machine learning step, in which a machine learning-basedtraining algorithm is used to train a personalized model that isevaluated against user measurements in known states. The machinelearning step feeds the refinement step with processed data andquestions, and in the refinement step the user answers whether themachine learning algorithm has correctly detected the user's state. Boththe machine learning step and the refinement step may provide data tothe optional detection and biofeedback steps (which may communicate witheach other).

Big data analysis may be performed to identify trends and detect newcorrelations over users and populations, together with other sources ofinformation, such as other wearable devices (e.g., smart watches, smartshirts, EEG headsets, smart earphones), mobile devices (e.g.,smartphone, laptop), and other sources of information (e.g., socialnetworks, search engines, bots, software agents, medical records, IoTdevices).

Various embodiments described herein involve an HMS that may beconnected, using wires and/or wirelessly, with a device carried by theuser and/or a non-wearable device. The HMS may include a battery, acomputer, sensors, and a transceiver.

FIG. 34a and FIG. 34b are schematic illustrations of possibleembodiments for computers (400, 410) that are able to realize one ormore of the embodiments discussed herein that include a “computer”. Thecomputer (400, 410) may be implemented in various ways, such as, but notlimited to, a server, a client, a personal computer, a network device, ahandheld device (e.g., a smartphone), an HMS (such as smart glasses, anaugmented reality system, and/or a virtual reality system), a computingdevice embedded in a wearable device (e.g., a smartwatch or a computerembedded in clothing), a computing device implanted in the human body,and/or any other computer form capable of executing a set of computerinstructions. Herein, an augmented reality system refers also to a mixedreality system. Further, references to a computer or processor includeany collection of one or more computers and/or processors (which may beat different locations) that individually or jointly execute one or moresets of computer instructions. For example, a first computer may beembedded in the HMS that communicates with a second computer embedded inthe user's smartphone that communicates over the Internet with a cloudcomputer.

The computer 400 includes one or more of the following components:processor 401, memory 402, computer readable medium 403, user interface404, communication interface 405, and bus 406. The computer 410 includesone or more of the following components: processor 411, memory 412, andcommunication interface 413.

Thermal measurements that are forwarded to a processor/computer mayinclude “raw” values that are essentially the same as the valuesmeasured by thermal cameras, and/or processed values that are the resultof applying some form of preprocessing and/or analysis to the rawvalues. Examples of methods that may be used to process the raw valuesinclude analog signal processing, digital signal processing, and variousforms of normalization, noise cancellation, and/or feature extraction.

Functionality of various embodiments may be implemented in hardware,software, firmware, or any combination thereof. If implemented at leastin part in software, implementing the functionality may involve acomputer program that includes one or more instructions or code storedor transmitted on a computer-readable medium and executed by one or moreprocessors. Computer-readable media may include computer-readablestorage media, which corresponds to a tangible medium such as datastorage media, or communication media including any medium thatfacilitates transfer of a computer program from one place to another.Computer-readable medium may be any media that can be accessed by one ormore computers to retrieve instructions, code, data, and/or datastructures for implementation of the described embodiments. A computerprogram product may include a computer-readable medium. In one example,the computer-readable medium 403 may include one or more of thefollowing: RAM, ROM, EEPROM, optical storage, magnetic storage, biologicstorage, flash memory, or any other medium that can store computerreadable data.

A computer program (also known as a program, software, softwareapplication, script, program code, or code) can be written in any formof programming language, including compiled or interpreted languages,declarative or procedural languages. The program can be deployed in anyform, including as a standalone program or as a module, component,subroutine, object, or another unit suitable for use in a computingenvironment. A computer program may correspond to a file in a filesystem, may be stored in a portion of a file that holds other programsor data, and/or may be stored in one or more files that may be dedicatedto the program. A computer program may be deployed to be executed on oneor more computers that are located at one or more sites that may beinterconnected by a communication network.

Computer-readable medium may include a single medium and/or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store one or more sets of instructions. Invarious embodiments, a computer program, and/or portions of a computerprogram, may be stored on a non-transitory computer-readable medium, andmay be updated and/or downloaded via a communication network, such asthe Internet. Optionally, the computer program may be downloaded from acentral repository, such as Apple App Store and/or Google Play.Optionally, the computer program may be downloaded from a repository,such as an open source and/or community run repository (e.g., GitHub).

At least some of the methods described herein are “computer-implementedmethods” that are implemented on a computer, such as the computer (400,410), by executing instructions on the processor (401, 411).Additionally, at least some of these instructions may be stored on anon-transitory computer-readable medium.

Herein, a direction of the optical axis of a VCAM or a CAM that hasfocusing optics is determined by the focusing optics, while thedirection of the optical axis of a CAM without focusing optics (such asa single pixel thermopile) is determined by the angle of maximumresponsivity of its sensor. When optics are utilized to takemeasurements with a CAM, then the term CAM includes the optics (e.g.,one or more lenses). In some embodiments, the optics of a CAM mayinclude one or more lenses made of a material suitable for the requiredwavelength, such as one or more of the following materials: CalciumFluoride, Gallium Arsenide, Germanium, Potassium Bromide, Sapphire,Silicon, Sodium Chloride, and Zinc Sulfide. In other embodiments, theCAM optics may include one or more diffractive optical elements, and/oror a combination of one or more diffractive optical elements and one ormore refractive optical elements.

When CAM includes an optical limiter/field limiter/FOV limiter (such asa thermopile sensor inside a standard TO-39 package with a window, or athermopile sensor with a polished metal field limiter), then the termCAM may also refer to the optical limiter. Depending on the context, theterm CAM may also refer to a readout circuit adjacent to CAM, and/or tothe housing that holds CAM.

Herein, references to thermal measurements in the context of calculatingvalues based on thermal measurements, generating feature values based onthermal measurements, or comparison of thermal measurements, relate tothe values of the thermal measurements (which are values of temperatureor values of temperature changes). Thus, a sentence in the form of“calculating based on TH_(ROI)” may be interpreted as “calculating basedon the values of TH_(ROI)”, and a sentence in the form of “comparingTH_(ROI1) and TH_(ROI2)” may be interpreted as “comparing values ofTH_(ROI1) and values of TH_(ROI1)”.

Depending on the embodiment, thermal measurements of an ROI (usuallydenoted TH_(ROI) or using a similar notation) may have various forms,such as time series, measurements taken according to a varying samplingfrequency, and/or measurements taken at irregular intervals. In someembodiments, thermal measurements may include various statistics of thetemperature measurements (T) and/or the changes to temperaturemeasurements (AT), such as minimum, maximum, and/or average values.Thermal measurements may be raw and/or processed values. When a thermalcamera has multiple sensing elements (pixels), the thermal measurementsmay include values corresponding to each of the pixels, and/or includevalues representing processing of the values of the pixels. The thermalmeasurements may be normalized, such as normalized with respect to abaseline (which is based on earlier thermal measurements), time of day,day in the month, type of activity being conducted by the user, and/orvarious environmental parameters (e.g., the environment's temperature,humidity, radiation level, etc.).

As used herein, references to “one embodiment” (and its variations) meanthat the feature being referred to may be included in at least oneembodiment of the invention. Moreover, separate references to “oneembodiment”, “some embodiments”, “another embodiment”, “still anotherembodiment”, etc., may refer to the same embodiment, may illustratedifferent aspects of an embodiment, and/or may refer to differentembodiments.

Some embodiments may be described using the verb “indicating”, theadjective “indicative”, and/or using variations thereof. Herein,sentences in the form of “X is indicative of Y” mean that X includesinformation correlated with Y, up to the case where X equals Y. Forexample, sentences in the form of “thermal measurements indicative of aphysiological response” mean that the thermal measurements includeinformation from which it is possible to infer the physiologicalresponse. Stating that “X indicates Y” or “X indicating Y” may beinterpreted as “X being indicative of Y”. Additionally, sentences in theform of “provide/receive an indication indicating whether X happened”may refer herein to any indication method, including but not limited to:sending/receiving a signal when X happened and not sending/receiving asignal when X did not happen, not sending/receiving a signal when Xhappened and sending/receiving a signal when X did not happen, and/orsending/receiving a first signal when X happened and sending/receiving asecond signal X did not happen.

Herein, “most” of something is defined as above 51% of the something(including 100% of the something). Both a “portion” of something and a“region” of something refer herein to a value between a fraction of thesomething and 100% of the something. For example, sentences in the formof a “portion of an area” may cover between 0.1% and 100% of the area.As another example, sentences in the form of a “region on the user'sforehead” may cover between the smallest area captured by a single pixel(such as 0.1% or 5% of the forehead) and 100% of the forehead. The word“region” refers to an open-ended claim language, and a camera said tocapture a specific region on the face may capture just a small part ofthe specific region, the entire specific region, and/or a portion of thespecific region together with additional region(s).

Sentences in the form of “angle greater than 20°” refer to absolutevalues (which may be +20° or −20° in this example), unless specificallyindicated, such as in a phrase having the form of “the optical axis ofCAM is 20° above/below the Frankfort horizontal plane” where it isclearly indicated that the CAM is pointed upwards/downwards. TheFrankfort horizontal plane is created by two lines from the superioraspects of the right/left external auditory canal to the most inferiorpoint of the right/left orbital rims.

The terms “comprises,” “comprising,” “includes,” “including,” “has,”“having”, or any other variation thereof, indicate an open-ended claimlanguage that does not exclude additional limitations. The “a” or “an”is employed to describe one or more, and the singular also includes theplural unless it is obvious that it is meant otherwise; for example,sentences in the form of “a CAM configured to take thermal measurementsof a region (TH_(ROI))” refers to one or more CAMs that take thermalmeasurements of one or more regions, including one CAM that takesthermal measurements of multiple regions; as another example, “acomputer” refers to one or more computers, such as a combination of awearable computer that operates together with a cloud computer.

The phrase “based on” is intended to mean “based, at least in part, on”.Additionally, stating that a value is calculated “based on X” andfollowing that, in a certain embodiment, that the value is calculated“also based on Y”, means that in the certain embodiment, the value iscalculated based on X and Y.

The terms “first”, “second” and so forth are to be interpreted merely asordinal designations, and shall not be limited in themselves. Apredetermined value is a fixed value and/or a value determined any timebefore performing a calculation that compares a certain value with thepredetermined value. A value is also considered to be a predeterminedvalue when the logic, used to determine whether a threshold thatutilizes the value is reached, is known before start performingcomputations to determine whether the threshold is reached.

The embodiments of the invention may include any variety of combinationsand/or integrations of the features of the embodiments described herein.Although some embodiments may depict serial operations, the embodimentsmay perform certain operations in parallel and/or in different ordersfrom those depicted. Moreover, the use of repeated reference numeralsand/or letters in the text and/or drawings is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed. Theembodiments are not limited in their applications to the order of stepsof the methods, or to details of implementation of the devices, set inthe description, drawings, or examples. Moreover, individual blocksillustrated in the figures may be functional in nature and therefore maynot necessarily correspond to discrete hardware elements.

Certain features of the embodiments, which may have been, for clarity,described in the context of separate embodiments, may also be providedin various combinations in a single embodiment. Conversely, variousfeatures of the embodiments, which may have been, for brevity, describedin the context of a single embodiment, may also be provided separatelyor in any suitable sub-combination. Embodiments described in conjunctionwith specific examples are presented by way of example, and notlimitation. Moreover, it is evident that many alternatives,modifications, and variations will be apparent to those skilled in theart. It is to be understood that other embodiments may be utilized andstructural changes may be made without departing from the scope of theembodiments. Accordingly, this disclosure is intended to embrace allsuch alternatives, modifications, and variations that fall within thespirit and scope of the appended claims and their equivalents.

We claim:
 1. A system configured to detect a physiological responsewhile taking into account consumption of a confounding substance,comprising: an inward-facing head-mounted thermal camera (CAM)configured to take thermal measurements of a region of interest(TH_(ROI)) on a user's face; and a computer configured to: receive anindication indicative of the user consuming a confounding substance thataffects TH_(ROI); and detect the physiological response, while theconsumed confounding substance affects TH_(ROI), based on TH_(ROI), theindication, and a model.
 2. The system of claim 1, wherein the model wastrained on: a first set of TH_(ROI) taken while the confoundingsubstance affected TH_(ROI), and a second set of TH_(ROI) taken whilethe confounding substance did not affect TH_(ROI).
 3. The system ofclaim 2, wherein CAM is located less than 10 cm from the user's face,and the confounding substance comprises at least one of an alcoholicbeverage, a medication, and a cigarette; and wherein the first andsecond sets are previous TH_(ROI) of the user.
 4. The system of claim 2,wherein the computer is configured to detect the physiological responsebased on first TH_(ROI), for which there is no indication indicatingthat the first TH_(ROI) were affected by a consumption of theconfounding substance, and the first TH_(ROI) reach a threshold; and thecomputer is configured not to detect the physiological response based onsecond TH_(ROI), for which there is an indication indicating that thesecond TH_(ROI) were affected by a consumption the confoundingsubstance, and the second TH_(ROI) also reach the threshold.
 5. Thesystem of claim 2, wherein the model is a machine learning-based model,and the computer is further configured to generate feature values basedon TH_(ROI) and the indication, and to utilize the machinelearning-based model to detect the physiological response based on thefeature values.
 6. The system of claim 2, wherein the confoundingsubstance comprises a medication, the indication is received from atleast one of a pill dispenser and a sensor-enabled pill, and theindication indicates that the user took a medication.
 7. The system ofclaim 2, wherein the confounding substance comprises an alcoholicbeverage, the indication is received from at least one of arefrigerator, a pantry, and a serving robot; and wherein the indicationindicates that the user took an alcoholic beverage.
 8. The system ofclaim 2, wherein the indication is received from a head-mountedvisible-light camera having in its field of view a volume that protrudesout of the user's mouth, and the computer is further configured toidentify the consuming of the confounding substance based on analysis ofimages taken by the head-mounted visible-light camera.
 9. The system ofclaim 2, wherein the indication is received from a device having aninternet-of-things (IoT) capability through which the indication isprovided to the computer.
 10. The system of claim 2, wherein theindication is received from a microphone configured to record the user;and the computer is further configured to identify the consuming of theconfounding substance utilizing a sound recognition algorithm operatedon a recording of the user.
 11. The system of claim 10, wherein thesound recognition algorithm comprises a speech recognition algorithmconfigured to identify words that are indicative of consuming theconfounding substance.
 12. The system of claim 2, wherein the indicationis received from a user interface configured to receive an input fromthe user or a third party about the consuming of the confoundingsubstance.
 13. The system of claim 2, wherein the region of interest(ROI) is on the forehead, and CAM: is physically coupled to aneyeglasses frame, located below the ROI, and does not occlude the ROI.14. The system of claim 2, wherein the region of interest (ROI) is onthe periorbital area, and CAM is located less than 10 cm from the face.15. The system of claim 2, wherein the region of interest (ROI) is onthe nose, and CAM: is physically coupled to an eyeglasses frame, andlocated less than 10 cm from the face.
 16. The system of claim 2,wherein the region of interest (ROI) is below the nostrils, and CAM: isphysically coupled to an eyeglasses frame, located above the ROI, anddoes not occlude the ROI.
 17. A method for detecting a physiologicalresponse while taking into account consumption of a confoundingsubstance, comprising: taking, using an inward-facing head-mountedthermal camera, thermal measurements of a region of interest (TH_(ROI))on a user's face; receiving an indication indicative of consuming aconfounding substance that affects TH_(ROI); and detecting thephysiological response, while the consumed confounding substance affectsTH_(ROI), based on TH_(ROI), the indication, and a model; wherein themodel was trained on: a first set of TH_(ROI) taken while theconfounding substance affected TH_(ROI), and a second set of TH_(ROI)taken while the confounding substance did not affect TH_(ROI).
 18. Themethod of claim 17, wherein the model is a machine learning-based model,and further comprising generating feature values based on TH_(ROI) andthe indication, and utilizing the machine learning-based model to detectthe physiological response based on the feature values.
 19. The methodof claim 17, further comprising generating the indication based onanalyzing images taken by a head-mounted visible-light camera having inits field of view at least a portion of a volume that protrudes out ofthe user's mouth.
 20. The method of claim 17, wherein the confoundingsubstance comprises at least one of: (i) a medication, wherein theindication is received from at least one of a pill dispenser and asensor-enabled pill, and (ii) an alcoholic beverage, wherein theindication is received from at least one of a refrigerator, a pantry,and a serving robot.